[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:18 dist-test-slave-6nmv krb5kdc[30020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:18 dist-test-slave-6nmv krb5kdc[30020](info): set up 2 sockets
Aug 10 22:56:18 dist-test-slave-6nmv krb5kdc[30020](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 22:56:19.878931 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.008s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:19 dist-test-slave-6nmv krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:19.997032 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35053
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:20.134162 30033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:20.134456 30033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:20.134531 30033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:20.138564 30033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:20.138643 30033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:20.138679 30033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:20.138717 30033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:20.138746 30033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:20.144069 30033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:20.145342 30033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:20.146556 30033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:20.159802 30043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:20.160843 30033 server_base.cc:1034] running on GCE node
W20260810 22:56:20.161274 30041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:20.161291 30040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:20.162230 30033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:20.163961 30033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:20.165141 30033 hybrid_clock.cc:648] HybridClock initialized: now 1786402580165125 us; error 25 us; skew 500 ppm
Aug 10 22:56:20 dist-test-slave-6nmv krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402580, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:20.168561 30033 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:20.169562 30033 webserver.cc:460] Webserver started at http://127.29.78.190:36173/ using document root <none> and password file <none>
I20260810 22:56:20.170372 30033 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:20.170456 30033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:20.170713 30033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:20.173439 30033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "56af11d69b5d45bcad5333bd6f096354"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "83a2570c3b2e074dc00c93fc8c6585f2"
server_key_iv: "29114916c280e30455d006eefe483a9a"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.174327 30033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "56af11d69b5d45bcad5333bd6f096354"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "83a2570c3b2e074dc00c93fc8c6585f2"
server_key_iv: "29114916c280e30455d006eefe483a9a"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.179606 30033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 22:56:20.182837 30050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:20.184331 30033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:56:20.184481 30033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "56af11d69b5d45bcad5333bd6f096354"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "83a2570c3b2e074dc00c93fc8c6585f2"
server_key_iv: "29114916c280e30455d006eefe483a9a"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.184569 30033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:20.200358 30033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:20.203533 30033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:20.203784 30033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:20.218508 30033 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35053
I20260810 22:56:20.218567 30108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35053 every 8 connection(s)
I20260810 22:56:20.219650 30033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:20.223775 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 30033
I20260810 22:56:20.223976 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:20.224498 30109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:20.226152 30010 external_mini_cluster.cc:1428] Setting key a9887d2611042d67ea26b9d6a64fafd8
I20260810 22:56:20.235028 30109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354: Bootstrap starting.
I20260810 22:56:20.239729 30109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:20.241451 30109 log.cc:824] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:20.244438 30109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354: No bootstrap required, opened a new log
I20260810 22:56:20.251245 30109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56af11d69b5d45bcad5333bd6f096354" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35053 } }
I20260810 22:56:20.251547 30109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:20.251724 30109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56af11d69b5d45bcad5333bd6f096354, State: Initialized, Role: FOLLOWER
Aug 10 22:56:20 dist-test-slave-6nmv krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402579, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:20.252517 30109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56af11d69b5d45bcad5333bd6f096354" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35053 } }
I20260810 22:56:20.252777 30109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:20.252941 30109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:20.253103 30109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:20.254264 30109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56af11d69b5d45bcad5333bd6f096354" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35053 } }
I20260810 22:56:20.254741 30109 leader_election.cc:302] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56af11d69b5d45bcad5333bd6f096354; no voters:
I20260810 22:56:20.255095 30109 leader_election.cc:288] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:20.255314 30116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:20.255673 30116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [term 1 LEADER]: Becoming Leader. State: Replica: 56af11d69b5d45bcad5333bd6f096354, State: Running, Role: LEADER
I20260810 22:56:20.256095 30116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56af11d69b5d45bcad5333bd6f096354" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35053 } }
I20260810 22:56:20.256702 30109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:20.259162 30118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56af11d69b5d45bcad5333bd6f096354. Latest consensus state: current_term: 1 leader_uuid: "56af11d69b5d45bcad5333bd6f096354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56af11d69b5d45bcad5333bd6f096354" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35053 } } }
I20260810 22:56:20.259374 30118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:20.260295 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:20.231227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.29.78.190:35053)
0810 22:56:20.231954 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:20.231982 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:20.233209 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 22:56:20.239539 (+ 6330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:20.239555 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:20.239906 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:20.240144 (+ 238us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:20.243845 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.244927 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.246732 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.247158 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.247263 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:20.252907 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:20.252920 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:20.252935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:20.252968 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:20.255414 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:20.256117 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:20.256123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:20.256128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:20.256273 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:20.256667 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:20.256670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:20.256671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:20.257464 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:20.257718 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:20.258633 (+ 915us) server_negotiation.cc:299] Negotiation successful
0810 22:56:20.258977 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":234,"threads_started":1}
I20260810 22:56:20.260551 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56af11d69b5d45bcad5333bd6f096354" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56af11d69b5d45bcad5333bd6f096354" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35053 } } }
I20260810 22:56:20.260679 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:20.261950 30126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:20.267442 30126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:20.276357 30126 catalog_manager.cc:1347] Generated new cluster ID: 6fe5a2fc71b34840ac4991bef8978990
I20260810 22:56:20.276430 30126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:20.298683 30126 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:20.299636 30126 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:20.307458 30126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56af11d69b5d45bcad5333bd6f096354: Generated new TSK 0
I20260810 22:56:20.308214 30126 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:20.351074 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35053
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:20.488615 30140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:20.488920 30140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:20.489013 30140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:20.492938 30140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:20.493038 30140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:20.493140 30140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:20.498339 30140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35053
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:20.499346 30140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:20.500372 30140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:20.510563 30147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:20.510514 30146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:20.510547 30149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:20.510983 30140 server_base.cc:1034] running on GCE node
I20260810 22:56:20.511310 30140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:20.511937 30140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:20.513095 30140 hybrid_clock.cc:648] HybridClock initialized: now 1786402580513068 us; error 40 us; skew 500 ppm
Aug 10 22:56:20 dist-test-slave-6nmv krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402580, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:20.516237 30140 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:20.517110 30140 webserver.cc:460] Webserver started at http://127.29.78.129:39605/ using document root <none> and password file <none>
I20260810 22:56:20.517637 30140 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:20.517726 30140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:20.517998 30140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:20.520147 30140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "9d354b7526324f5d824ac476d6c5c8ea"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "a1ccae92636d6af16d367fa07deef62c"
server_key_iv: "86f9479c508c93b99b4029381024e707"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.520772 30140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "9d354b7526324f5d824ac476d6c5c8ea"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "a1ccae92636d6af16d367fa07deef62c"
server_key_iv: "86f9479c508c93b99b4029381024e707"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.524523 30140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 22:56:20.526986 30157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:20.528064 30140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:20.528192 30140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "9d354b7526324f5d824ac476d6c5c8ea"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "a1ccae92636d6af16d367fa07deef62c"
server_key_iv: "86f9479c508c93b99b4029381024e707"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.528303 30140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:20.541927 30140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:20.544793 30140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:20.545051 30140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:20.545877 30140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:20.546991 30140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:20.547061 30140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:20.547128 30140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:20.547173 30140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:20.565516 30140 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:42969
I20260810 22:56:20.565575 30291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:42969 every 8 connection(s)
I20260810 22:56:20.566479 30140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:20.568554 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 30140
I20260810 22:56:20.568656 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:20.568915 30010 external_mini_cluster.cc:1428] Setting key 8be684b8494740db471c558a57c4dc06
Aug 10 22:56:20 dist-test-slave-6nmv krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402580, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:20.582482 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:20.568527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:51623 (local address 127.29.78.190:35053)
0810 22:56:20.568670 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:20.568675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:20.569887 (+ 1212us) server_negotiation.cc:407] Connection header received
0810 22:56:20.571671 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:20.571675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:20.571730 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:20.571833 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:20.573454 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.574046 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.575810 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.576027 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.576154 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:20.578843 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:20.578846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:20.578847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:20.578875 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:20.580301 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:20.580887 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:20.580890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:20.580892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:20.580972 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:20.581296 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:20.581301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:20.581303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:20.581481 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:20.581581 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:20.582187 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 22:56:20.582315 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:56:20.583371 30294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:20.568989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35053 (local address 127.29.78.129:51623)
0810 22:56:20.569721 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:20.569762 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:20.571437 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:20.572019 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:20.572026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:20.572432 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:20.573266 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:20.573290 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.574187 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.574192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:20.575683 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:20.575697 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.576221 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.576228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:20.576478 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:20.577229 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:20.577246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:20.578676 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:20.580424 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:20.580430 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:20.580445 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:20.580775 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:20.581067 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:20.581070 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:20.581071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:20.581205 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:20.581593 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:20.581600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:20.581935 (+ 335us) client_negotiation.cc:769] Sending connection context
0810 22:56:20.582144 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 22:56:20.582473 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":292,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 22:56:20.584805 30292 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35053
I20260810 22:56:20.585079 30292 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:20.585672 30292 heartbeater.cc:511] Master 127.29.78.190:35053 requested a full tablet report, sending...
I20260810 22:56:20.587786 30067 ts_manager.cc:194] Registered new tserver with Master: 9d354b7526324f5d824ac476d6c5c8ea (127.29.78.129:42969)
I20260810 22:56:20.589246 30067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:51623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:20.600818 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35053
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:20.737383 30298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:20.737602 30298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:20.737641 30298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:20.741189 30298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:20.741233 30298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:20.741317 30298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:20.746172 30298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35053
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:20.747234 30298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:20.748265 30298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:20.757895 30303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:20.757902 30306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:20.757942 30304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:20.758252 30298 server_base.cc:1034] running on GCE node
I20260810 22:56:20.758684 30298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:20.759356 30298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:20.760545 30298 hybrid_clock.cc:648] HybridClock initialized: now 1786402580760512 us; error 45 us; skew 500 ppm
Aug 10 22:56:20 dist-test-slave-6nmv krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402580, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:20.763621 30298 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:20.764497 30298 webserver.cc:460] Webserver started at http://127.29.78.130:44301/ using document root <none> and password file <none>
I20260810 22:56:20.765060 30298 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:20.765172 30298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:20.765417 30298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:20.767555 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "c8778ed17ff048ab85cb93ee29184b9d"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "0d2f13ad2af8ac68580a2a3fbba812d5"
server_key_iv: "5c427ea6748ce556ac1a7ac3c2a112cb"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.768184 30298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "c8778ed17ff048ab85cb93ee29184b9d"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "0d2f13ad2af8ac68580a2a3fbba812d5"
server_key_iv: "5c427ea6748ce556ac1a7ac3c2a112cb"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.771994 30298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 22:56:20.774420 30314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:20.775578 30298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:20.775715 30298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "c8778ed17ff048ab85cb93ee29184b9d"
format_stamp: "Formatted at 2026-08-10 22:56:20 on dist-test-slave-6nmv"
server_key: "0d2f13ad2af8ac68580a2a3fbba812d5"
server_key_iv: "5c427ea6748ce556ac1a7ac3c2a112cb"
server_key_version: "encryptionkey@0"
I20260810 22:56:20.775825 30298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:20.792112 30298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:20.796096 30298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:20.796370 30298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:20.797008 30298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:20.797864 30298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:20.797930 30298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:20.797983 30298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:20.798009 30298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:20.815316 30298 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:34489
I20260810 22:56:20.815405 30442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:34489 every 8 connection(s)
I20260810 22:56:20.816262 30298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:20.817749 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 30298
I20260810 22:56:20.817914 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:20.818189 30010 external_mini_cluster.cc:1428] Setting key 2705398700d2864272200015918238ff
Aug 10 22:56:20 dist-test-slave-6nmv krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402580, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:56:20.830982 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:20.818131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44847 (local address 127.29.78.190:35053)
0810 22:56:20.818255 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:20.818260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:20.819433 (+ 1173us) server_negotiation.cc:407] Connection header received
0810 22:56:20.820967 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:20.820973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:20.821032 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:20.821118 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:20.822535 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.823347 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.824566 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.824791 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.824898 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:20.827554 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:20.827562 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:20.827564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:20.827586 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:20.829037 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:20.829518 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:20.829523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:20.829528 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:20.829598 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:20.829965 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:20.829969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:20.829971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:20.830145 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:20.830254 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:20.830714 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 22:56:20.830827 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:56:20.831734 30446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:20.818606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35053 (local address 127.29.78.130:44847)
0810 22:56:20.819253 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:20.819305 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:20.820749 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:20.821256 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:20.821262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:20.821767 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:20.822394 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:20.822408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.823459 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.823461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:20.824438 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:20.824448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:20.824946 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:20.824950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:20.825176 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:20.825900 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:20.825916 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:20.827398 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:20.829166 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:20.829171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:20.829183 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:20.829414 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:20.829685 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:20.829689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:20.829691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:20.829883 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:20.830252 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:20.830258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:20.830495 (+ 237us) client_negotiation.cc:769] Sending connection context
0810 22:56:20.830660 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 22:56:20.830939 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":229,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260810 22:56:20.833041 30443 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35053
I20260810 22:56:20.833323 30443 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:20.833766 30443 heartbeater.cc:511] Master 127.29.78.190:35053 requested a full tablet report, sending...
I20260810 22:56:20.834817 30067 ts_manager.cc:194] Registered new tserver with Master: c8778ed17ff048ab85cb93ee29184b9d (127.29.78.130:34489)
I20260810 22:56:20.835376 30067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:44847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:20.849023 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35053
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:20.996304 30449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:20.996671 30449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:20.996721 30449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:21.000600 30449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:21.000653 30449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:21.000708 30449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:21.005677 30449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35053
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:21.006646 30449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:21.007534 30449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:21.017297 30455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:21.017349 30458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:21.017298 30456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:21.018003 30449 server_base.cc:1034] running on GCE node
I20260810 22:56:21.018373 30449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:21.018986 30449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:21.020179 30449 hybrid_clock.cc:648] HybridClock initialized: now 1786402581020149 us; error 48 us; skew 500 ppm
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402581, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:21.023521 30449 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:21.024344 30449 webserver.cc:460] Webserver started at http://127.29.78.131:42687/ using document root <none> and password file <none>
I20260810 22:56:21.024814 30449 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:21.024869 30449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:21.025075 30449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:21.027206 30449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "cbe76b1473254865b4814dbedfe3a066"
format_stamp: "Formatted at 2026-08-10 22:56:21 on dist-test-slave-6nmv"
server_key: "8ef8ffa3d85e7ebb97e3682219f80460"
server_key_iv: "915a84d94c13b415db97fee9446c71ff"
server_key_version: "encryptionkey@0"
I20260810 22:56:21.027817 30449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "cbe76b1473254865b4814dbedfe3a066"
format_stamp: "Formatted at 2026-08-10 22:56:21 on dist-test-slave-6nmv"
server_key: "8ef8ffa3d85e7ebb97e3682219f80460"
server_key_iv: "915a84d94c13b415db97fee9446c71ff"
server_key_version: "encryptionkey@0"
I20260810 22:56:21.031839 30449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:56:21.034487 30467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:21.035727 30449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:21.035825 30449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "cbe76b1473254865b4814dbedfe3a066"
format_stamp: "Formatted at 2026-08-10 22:56:21 on dist-test-slave-6nmv"
server_key: "8ef8ffa3d85e7ebb97e3682219f80460"
server_key_iv: "915a84d94c13b415db97fee9446c71ff"
server_key_version: "encryptionkey@0"
I20260810 22:56:21.035913 30449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:21.073583 30449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:21.077575 30449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:21.077757 30449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:21.078497 30449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:21.079545 30449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:21.079596 30449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:21.079645 30449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:21.079694 30449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:21.096863 30449 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:35437
I20260810 22:56:21.097012 30597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:35437 every 8 connection(s)
I20260810 22:56:21.097787 30449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:21.106145 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 30449
I20260810 22:56:21.106271 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:21.106565 30010 external_mini_cluster.cc:1428] Setting key a4d2d589f2745491bdc9420833d22e4a
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402581, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:21.112165 30112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:21.099555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60491 (local address 127.29.78.190:35053)
0810 22:56:21.099691 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:21.099694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:21.100597 (+ 903us) server_negotiation.cc:407] Connection header received
0810 22:56:21.102162 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:21.102165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:21.102232 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:21.102323 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:21.103739 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.104260 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.105421 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.105626 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.105734 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:21.108751 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:21.108754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:21.108755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:21.108775 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:21.110424 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:21.110902 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:21.110904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:21.110905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:21.110969 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:21.111267 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:21.111268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:21.111269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:21.111420 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:21.111487 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:21.111971 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 22:56:21.112062 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:56:21.113044 30600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:21.099880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35053 (local address 127.29.78.131:60491)
0810 22:56:21.100448 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:21.100491 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:21.101954 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:21.102485 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:21.102491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:21.103004 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:21.103582 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:21.103598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.104392 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.104399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:21.105292 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:21.105302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.105769 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.105773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:21.106014 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:21.106681 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:21.106703 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:21.108598 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:21.110532 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:21.110536 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:21.110546 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:21.110799 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:21.111060 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:21.111062 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:21.111063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:21.111189 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:21.111515 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:21.111519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:21.111767 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 22:56:21.111933 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 22:56:21.112153 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":179,"threads_started":1}
I20260810 22:56:21.114257 30598 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35053
I20260810 22:56:21.114472 30598 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:21.114916 30598 heartbeater.cc:511] Master 127.29.78.190:35053 requested a full tablet report, sending...
I20260810 22:56:21.115898 30067 ts_manager.cc:194] Registered new tserver with Master: cbe76b1473254865b4814dbedfe3a066 (127.29.78.131:35437)
I20260810 22:56:21.116439 30067 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:60491
I20260810 22:56:21.121022 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402581, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:21.148424 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:21.138673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50082 (local address 127.29.78.129:42969)
0810 22:56:21.139144 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:21.139150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:21.139173 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:56:21.139219 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:21.139224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:21.139421 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:21.139575 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:21.140380 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.141276 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.142166 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.142586 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.142671 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:21.144999 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:21.145007 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:21.145019 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:21.145040 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:21.146880 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:21.147235 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:21.147240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:21.147241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:21.147313 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:21.147530 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:21.147533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:21.147534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:21.147739 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:21.147898 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:21.148015 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 22:56:21.148167 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":163,"threads_started":1}
W20260810 22:56:21.149528 30187 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50082
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:21.176427 30605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:21.167992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50090 (local address 127.29.78.129:42969)
0810 22:56:21.168130 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:21.168134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:21.168290 (+ 156us) server_negotiation.cc:407] Connection header received
0810 22:56:21.168506 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:21.168508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:21.168548 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:21.168627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:21.169229 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.169678 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.170505 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:21.170712 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:21.170835 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:21.173255 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:21.173258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:21.173260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:21.173285 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:21.174760 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:21.175195 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:21.175199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:21.175201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:21.175285 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:21.175572 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:21.175575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:21.175577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:21.175748 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:21.175892 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:21.176133 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 22:56:21.176262 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:56:21.177513 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 30140
I20260810 22:56:21.187842 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 30298
I20260810 22:56:21.195716 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 30449
I20260810 22:56:21.203292 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 30033
2026-08-10T22:56:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2390 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30614](info): set up 2 sockets
Aug 10 22:56:21 dist-test-slave-6nmv krb5kdc[30614](info): commencing operation
krb5kdc: starting...
W20260810 22:56:22.204504 30010 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:22 dist-test-slave-6nmv krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:22.318837 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40643
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:22.436530 30628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:22.436908 30628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:22.436982 30628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:22.440659 30628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:22.440737 30628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:22.440769 30628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:22.440828 30628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:22.440871 30628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:22.445627 30628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:22.446630 30628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:22.447706 30628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:22.456302 30634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:22.456310 30636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:22.456310 30633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:22.456681 30628 server_base.cc:1034] running on GCE node
I20260810 22:56:22.457213 30628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:22.458361 30628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:22.459547 30628 hybrid_clock.cc:648] HybridClock initialized: now 1786402582459524 us; error 46 us; skew 500 ppm
Aug 10 22:56:22 dist-test-slave-6nmv krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:22.462095 30628 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:22.462898 30628 webserver.cc:460] Webserver started at http://127.29.78.190:41539/ using document root <none> and password file <none>
I20260810 22:56:22.463461 30628 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:22.463563 30628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:22.463793 30628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:22.465929 30628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "4b6e5ca565944c2051844670eb37302b"
server_key_iv: "6543b697a99debc802d0347c32f71dfa"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.466457 30628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "4b6e5ca565944c2051844670eb37302b"
server_key_iv: "6543b697a99debc802d0347c32f71dfa"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.470075 30628 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:56:22.472333 30643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:22.473486 30628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:22.473632 30628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "4b6e5ca565944c2051844670eb37302b"
server_key_iv: "6543b697a99debc802d0347c32f71dfa"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.473737 30628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:22.492161 30628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:22.494753 30628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:22.494923 30628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:22.501387 30628 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40643
I20260810 22:56:22.501451 30706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40643 every 8 connection(s)
I20260810 22:56:22.502274 30628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:22.505035 30707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:22.505236 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 30628
I20260810 22:56:22.505343 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:22.505618 30010 external_mini_cluster.cc:1428] Setting key 6144768f4fbe660a7bae6c5ac11d1a01
I20260810 22:56:22.511719 30707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f: Bootstrap starting.
Aug 10 22:56:22 dist-test-slave-6nmv krb5kdc[30614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:22.514582 30707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:22.515296 30707 log.cc:824] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:22.516979 30707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f: No bootstrap required, opened a new log
I20260810 22:56:22.519105 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:22.506952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.29.78.190:40643)
0810 22:56:22.507340 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:22.507354 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:22.507386 (+ 32us) server_negotiation.cc:407] Connection header received
0810 22:56:22.508101 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:22.508118 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:22.508429 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:22.508692 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:22.509554 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:22.510548 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:22.511605 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:22.512126 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:22.512238 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:22.515002 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:22.515006 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:22.515017 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:22.515040 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:22.516635 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:22.517057 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:22.517079 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:22.517085 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:22.517199 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:22.517452 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:22.517456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:22.517457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:22.517785 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:22.518010 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:22.518247 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 22:56:22.518493 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":128,"threads_started":1}
I20260810 22:56:22.520653 30707 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40643 } }
I20260810 22:56:22.520895 30707 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:22.520941 30707 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcaaf58d1ecf4cf59c00cd0b644b065f, State: Initialized, Role: FOLLOWER
I20260810 22:56:22.521404 30707 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40643 } }
I20260810 22:56:22.521562 30707 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:22.521616 30707 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:22.521682 30707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:22.522495 30707 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40643 } }
I20260810 22:56:22.522773 30707 leader_election.cc:302] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcaaf58d1ecf4cf59c00cd0b644b065f; no voters:
I20260810 22:56:22.523041 30707 leader_election.cc:288] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:22.523177 30712 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:22.523411 30712 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [term 1 LEADER]: Becoming Leader. State: Replica: fcaaf58d1ecf4cf59c00cd0b644b065f, State: Running, Role: LEADER
I20260810 22:56:22.523697 30712 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40643 } }
I20260810 22:56:22.524241 30707 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:22.525177 30713 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40643 } } }
I20260810 22:56:22.525291 30713 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:22.525595 30714 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcaaf58d1ecf4cf59c00cd0b644b065f. Latest consensus state: current_term: 1 leader_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcaaf58d1ecf4cf59c00cd0b644b065f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40643 } } }
I20260810 22:56:22.525708 30714 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f [sys.catalog]: This master's current role is: LEADER
W20260810 22:56:22.528674 30728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:56:22.528759 30728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:56:22.528831 30721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:22.529543 30721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:22.535045 30721 catalog_manager.cc:1347] Generated new cluster ID: 639ef51804d64f088b6c9ac99605e294
I20260810 22:56:22.535109 30721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:22.541627 30721 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:22.542696 30721 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:22.556425 30721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcaaf58d1ecf4cf59c00cd0b644b065f: Generated new TSK 0
I20260810 22:56:22.557052 30721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:22.585312 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40643
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:22.691653 30734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:22.691875 30734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:22.691916 30734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:22.695559 30734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:22.695601 30734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:22.695650 30734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:22.700126 30734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:22.700927 30734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:22.701756 30734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:22.710117 30740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:22.710105 30744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:22.710100 30741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:22.710384 30734 server_base.cc:1034] running on GCE node
I20260810 22:56:22.710786 30734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:22.711382 30734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:22.712536 30734 hybrid_clock.cc:648] HybridClock initialized: now 1786402582712509 us; error 37 us; skew 500 ppm
Aug 10 22:56:22 dist-test-slave-6nmv krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:22.720065 30734 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:22.721124 30734 webserver.cc:460] Webserver started at http://127.29.78.129:43337/ using document root <none> and password file <none>
I20260810 22:56:22.721678 30734 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:22.721778 30734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:22.722028 30734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:22.724117 30734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "5282abfc11b7467099b22a3f53a1cdf7"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "178d6b3d5c38be54e214b708611bf2bf"
server_key_iv: "9d31f501d248e11cc20e9a72d6541b77"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.724967 30734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "5282abfc11b7467099b22a3f53a1cdf7"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "178d6b3d5c38be54e214b708611bf2bf"
server_key_iv: "9d31f501d248e11cc20e9a72d6541b77"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.728708 30734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 22:56:22.730927 30751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:22.732137 30734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:22.732278 30734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "5282abfc11b7467099b22a3f53a1cdf7"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "178d6b3d5c38be54e214b708611bf2bf"
server_key_iv: "9d31f501d248e11cc20e9a72d6541b77"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.732385 30734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:22.758472 30734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:22.761166 30734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:22.761373 30734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:22.761993 30734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:22.762789 30734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:22.762880 30734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:22.763000 30734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:22.763049 30734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:22.772150 30734 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:46303
I20260810 22:56:22.772188 30875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:46303 every 8 connection(s)
I20260810 22:56:22.773074 30734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:22.781282 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 30734
I20260810 22:56:22.781394 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:22.781715 30010 external_mini_cluster.cc:1428] Setting key 3da741177612947ec83e9d224b31d895
Aug 10 22:56:22 dist-test-slave-6nmv krb5kdc[30614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:22.786917 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:22.774875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:49195 (local address 127.29.78.190:40643)
0810 22:56:22.775009 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:22.775012 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:22.775825 (+ 813us) server_negotiation.cc:407] Connection header received
0810 22:56:22.776617 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:22.776620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:22.776676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:22.776783 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:22.778314 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:22.778849 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:22.779876 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:22.780083 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:22.780157 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:22.782908 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:22.782911 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:22.782913 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:22.782937 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:22.784713 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:22.785199 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:22.785205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:22.785209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:22.785323 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:22.785730 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:22.785735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:22.785738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:22.785966 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:22.786084 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:22.786631 (+ 547us) server_negotiation.cc:299] Negotiation successful
0810 22:56:22.786756 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:22.787673 30879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:22.775214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40643 (local address 127.29.78.129:49195)
0810 22:56:22.775672 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:22.775711 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:22.776414 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:22.776957 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:22.776967 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:22.777487 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:22.778168 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:22.778183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:22.778959 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:22.778961 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:22.779742 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:22.779750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:22.780204 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:22.780206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:22.780325 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:22.780886 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:22.780900 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:22.782728 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:22.784834 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:22.784839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:22.784850 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:22.785102 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:22.785435 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:22.785438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:22.785440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:22.785579 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:22.786095 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:22.786103 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:22.786376 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 22:56:22.786577 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 22:56:22.786883 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260810 22:56:22.789022 30877 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40643
I20260810 22:56:22.789255 30877 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:22.789713 30877 heartbeater.cc:511] Master 127.29.78.190:40643 requested a full tablet report, sending...
I20260810 22:56:22.791379 30666 ts_manager.cc:194] Registered new tserver with Master: 5282abfc11b7467099b22a3f53a1cdf7 (127.29.78.129:46303)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:22.793004 30666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:49195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:22.810331 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40643
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:22.917167 30884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:22.917384 30884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:22.917459 30884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:22.920900 30884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:22.920948 30884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:22.921015 30884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:22.926211 30884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:22.927150 30884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:22.928071 30884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:22.937366 30892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:22.937492 30884 server_base.cc:1034] running on GCE node
W20260810 22:56:22.937422 30894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:22.937327 30891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:22.938050 30884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:22.938568 30884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:22.939751 30884 hybrid_clock.cc:648] HybridClock initialized: now 1786402582939736 us; error 24 us; skew 500 ppm
Aug 10 22:56:22 dist-test-slave-6nmv krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:22.942319 30884 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:22.943153 30884 webserver.cc:460] Webserver started at http://127.29.78.130:46355/ using document root <none> and password file <none>
I20260810 22:56:22.943645 30884 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:22.943699 30884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:22.943851 30884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:22.945772 30884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "ec2abf5597c040219a123809c132ddce"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "0ac0782daa3fc13cd2ab5d1054b15a9f"
server_key_iv: "276c98ce171e884f8ce7b45e43b0c3df"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.946350 30884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "ec2abf5597c040219a123809c132ddce"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "0ac0782daa3fc13cd2ab5d1054b15a9f"
server_key_iv: "276c98ce171e884f8ce7b45e43b0c3df"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.949626 30884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:22.951820 30902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:22.952945 30884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:22.953078 30884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "ec2abf5597c040219a123809c132ddce"
format_stamp: "Formatted at 2026-08-10 22:56:22 on dist-test-slave-6nmv"
server_key: "0ac0782daa3fc13cd2ab5d1054b15a9f"
server_key_iv: "276c98ce171e884f8ce7b45e43b0c3df"
server_key_version: "encryptionkey@0"
I20260810 22:56:22.953174 30884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:22.983875 30884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:22.986820 30884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:22.987025 30884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:22.987625 30884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:22.988456 30884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:22.988502 30884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:22.988569 30884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:22.988605 30884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:22.997629 30884 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:41397
I20260810 22:56:22.997655 31025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:41397 every 8 connection(s)
I20260810 22:56:22.998517 30884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:23.006667 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 30884
I20260810 22:56:23.006798 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:23.007129 30010 external_mini_cluster.cc:1428] Setting key 20ea52078015eb16f881773a7e9b70b5
I20260810 22:56:23.011384 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.000193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49557 (local address 127.29.78.190:40643)
0810 22:56:23.000320 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.000331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.001055 (+ 724us) server_negotiation.cc:407] Connection header received
0810 22:56:23.001934 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.001938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.002007 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.002091 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:23.003217 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.003757 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.004829 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.005040 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.005162 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.007633 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:23.007637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:23.007640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:23.007669 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:23.009179 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.009784 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.009787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.009788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.009921 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.010321 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.010323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.010324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.010483 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:23.010554 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.011113 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.011227 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:56:23.012135 31031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.000504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40643 (local address 127.29.78.130:49557)
0810 22:56:23.000913 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:23.000952 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:23.001683 (+ 731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:23.002273 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:23.002279 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:23.002627 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:23.003072 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.003084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.003884 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.003887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.004713 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.004722 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.005197 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.005201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.005361 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.005936 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:23.005951 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:23.007449 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:23.009290 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:23.009297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:23.009311 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:23.009691 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:23.010029 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:23.010035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:23.010039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:23.010231 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:23.010577 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:23.010585 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:23.010875 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 22:56:23.011069 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 22:56:23.011342 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260810 22:56:23.013235 31027 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40643
I20260810 22:56:23.013475 31027 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:23.014011 31027 heartbeater.cc:511] Master 127.29.78.190:40643 requested a full tablet report, sending...
I20260810 22:56:23.015045 30666 ts_manager.cc:194] Registered new tserver with Master: ec2abf5597c040219a123809c132ddce (127.29.78.130:41397)
I20260810 22:56:23.015780 30666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:49557
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:23.036407 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40643
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:23.141196 31034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:23.141460 31034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:23.141503 31034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:23.145737 31034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:23.145787 31034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:23.145891 31034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:23.150529 31034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40643
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:23.151376 31034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:23.152318 31034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:23.161151 31044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:23.161221 31034 server_base.cc:1034] running on GCE node
W20260810 22:56:23.161171 31041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:23.161145 31042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:23.161688 31034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:23.162285 31034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:23.163466 31034 hybrid_clock.cc:648] HybridClock initialized: now 1786402583163452 us; error 29 us; skew 500 ppm
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402583, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:23.166018 31034 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:23.166822 31034 webserver.cc:460] Webserver started at http://127.29.78.131:44263/ using document root <none> and password file <none>
I20260810 22:56:23.167362 31034 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:23.167447 31034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:23.167654 31034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:23.169605 31034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "65db365f572c4d4783856596f522815d"
format_stamp: "Formatted at 2026-08-10 22:56:23 on dist-test-slave-6nmv"
server_key: "0733994320ee144a38818e877b5840d7"
server_key_iv: "6943c131f7b332a49a25ee1616ecbe92"
server_key_version: "encryptionkey@0"
I20260810 22:56:23.170240 31034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "65db365f572c4d4783856596f522815d"
format_stamp: "Formatted at 2026-08-10 22:56:23 on dist-test-slave-6nmv"
server_key: "0733994320ee144a38818e877b5840d7"
server_key_iv: "6943c131f7b332a49a25ee1616ecbe92"
server_key_version: "encryptionkey@0"
I20260810 22:56:23.173951 31034 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:56:23.176182 31054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:23.177312 31034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:23.177433 31034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "65db365f572c4d4783856596f522815d"
format_stamp: "Formatted at 2026-08-10 22:56:23 on dist-test-slave-6nmv"
server_key: "0733994320ee144a38818e877b5840d7"
server_key_iv: "6943c131f7b332a49a25ee1616ecbe92"
server_key_version: "encryptionkey@0"
I20260810 22:56:23.177527 31034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:23.192441 31034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:23.195020 31034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:23.195183 31034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:23.195647 31034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:23.196321 31034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:23.196381 31034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:23.196436 31034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:23.196472 31034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:23.207682 31034 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:45265
I20260810 22:56:23.207708 31177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:45265 every 8 connection(s)
I20260810 22:56:23.208478 31034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:23.212423 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 31034
I20260810 22:56:23.212525 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:23.212805 30010 external_mini_cluster.cc:1428] Setting key 2d19b3690ac43e6012aba4ad51726afd
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402583, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:23.221930 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.210342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38033 (local address 127.29.78.190:40643)
0810 22:56:23.210504 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.210509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.211559 (+ 1050us) server_negotiation.cc:407] Connection header received
0810 22:56:23.212724 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.212728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.212786 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.212899 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:23.214561 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.215110 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.216193 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.216385 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.216484 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.218940 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:23.218942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:23.218944 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:23.218965 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:23.220352 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.220748 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.220750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.220751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.220818 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.221074 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.221076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.221077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.221224 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:23.221292 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.221673 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.221755 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:56:23.222585 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.210702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40643 (local address 127.29.78.131:38033)
0810 22:56:23.211394 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:23.211449 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:23.212432 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:23.213094 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:23.213101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:23.213705 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:23.214420 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.214435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.215223 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.215226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.216077 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.216086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.216517 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.216521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.216752 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.217290 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:23.217304 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:23.218809 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:23.220441 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:23.220444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:23.220452 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:23.220671 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:23.220888 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:23.220890 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:23.220891 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:23.221002 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:23.221301 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:23.221305 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:23.221488 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 22:56:23.221630 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 22:56:23.221893 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":115,"threads_started":1}
I20260810 22:56:23.223683 31180 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40643
I20260810 22:56:23.223862 31180 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:23.224373 31180 heartbeater.cc:511] Master 127.29.78.190:40643 requested a full tablet report, sending...
I20260810 22:56:23.225296 30666 ts_manager.cc:194] Registered new tserver with Master: 65db365f572c4d4783856596f522815d (127.29.78.131:45265)
I20260810 22:56:23.226009 30666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:38033
I20260810 22:56:23.226430 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:56:23.236975 30710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.230486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.29.78.190:40643)
0810 22:56:23.230631 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.230635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.230807 (+ 172us) server_negotiation.cc:407] Connection header received
0810 22:56:23.230880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.230883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.230918 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.230976 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:23.231528 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.232000 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.232726 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.232916 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.233006 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.234262 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:23.234265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:23.234267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:23.234286 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:23.235464 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.235815 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.235817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.235818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.235883 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.236157 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.236159 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.236160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.236358 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:23.236479 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.236693 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.236774 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:56:23.241796 30666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:23.244993 30666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:23.261286 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.255550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:41397 (local address 127.0.0.1:51708)
0810 22:56:23.256936 (+ 1386us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:23.256950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:23.257082 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:23.257677 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:23.257680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:23.257698 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:23.257930 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.257939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.259174 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.259177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.259911 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.259920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.260723 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.260726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.260847 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.260888 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 22:56:23.261005 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 22:56:23.261121 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1326,"thread_start_us":94,"threads_started":1}
I20260810 22:56:23.261547 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.255769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.29.78.131:45265)
0810 22:56:23.256556 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.256573 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.256593 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:56:23.256690 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.256695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.256881 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.257362 (+ 481us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:23.257577 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.258839 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.259959 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.260910 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.261013 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.261149 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.261279 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.261365 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":222,"threads_started":1}
I20260810 22:56:23.261857 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.255749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.29.78.130:41397)
0810 22:56:23.256678 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.256718 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.257188 (+ 470us) server_negotiation.cc:407] Connection header received
0810 22:56:23.257274 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.257280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.257455 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.257681 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:23.258040 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.258925 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.260038 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.260593 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.260722 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.260926 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.261505 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.261615 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":161,"threads_started":1}
I20260810 22:56:23.261482 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.254954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45265 (local address 127.0.0.1:37074)
0810 22:56:23.256028 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:23.256075 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:23.256213 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:23.257026 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:23.257029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:23.257046 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:23.257193 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.257200 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.259054 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.259057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.259849 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.259858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.261033 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.261036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.261129 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.261151 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:56:23.261231 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:56:23.261286 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":30,"spinlock_wait_cycles":12800,"thread_start_us":148,"threads_started":1}
I20260810 22:56:23.262890 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.254752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46303 (local address 127.0.0.1:41410)
0810 22:56:23.256029 (+ 1277us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:23.256074 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:23.256213 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:23.256905 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:23.256912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:23.256936 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:23.257182 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.257194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.258916 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.258928 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.260974 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.260985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.262444 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.262448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.262530 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.262553 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:56:23.262641 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:56:23.262728 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1080,"mutex_wait_us":4,"spinlock_wait_cycles":4864,"thread_start_us":68,"threads_started":1}
I20260810 22:56:23.263018 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.255331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.29.78.129:46303)
0810 22:56:23.255805 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.255866 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.256364 (+ 498us) server_negotiation.cc:407] Connection header received
0810 22:56:23.256448 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.256470 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.256686 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.256907 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:23.257314 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.258703 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.261106 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.262301 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.262395 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.262515 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.262697 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.262788 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":155,"threads_started":1}
I20260810 22:56:23.265007 31107 tablet_service.cc:1467] Processing CreateTablet for tablet 8b55695c08904e8f9aafd3054e497589 (DEFAULT_TABLE table=test-table [id=a711bcd3e43a4ad29f8456828bb4a63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:23.265010 30956 tablet_service.cc:1467] Processing CreateTablet for tablet 8b55695c08904e8f9aafd3054e497589 (DEFAULT_TABLE table=test-table [id=a711bcd3e43a4ad29f8456828bb4a63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:23.266139 30956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b55695c08904e8f9aafd3054e497589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:23.267753 30804 tablet_service.cc:1467] Processing CreateTablet for tablet 8b55695c08904e8f9aafd3054e497589 (DEFAULT_TABLE table=test-table [id=a711bcd3e43a4ad29f8456828bb4a63e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:23.268510 31107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b55695c08904e8f9aafd3054e497589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:23.268685 30804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b55695c08904e8f9aafd3054e497589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:23.274360 31200 tablet_bootstrap.cc:492] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce: Bootstrap starting.
I20260810 22:56:23.277150 31201 tablet_bootstrap.cc:492] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d: Bootstrap starting.
I20260810 22:56:23.277902 31202 tablet_bootstrap.cc:492] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7: Bootstrap starting.
I20260810 22:56:23.278098 31200 tablet_bootstrap.cc:654] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:23.279803 31200 log.cc:824] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:23.281276 31202 tablet_bootstrap.cc:654] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:23.282135 31200 tablet_bootstrap.cc:492] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce: No bootstrap required, opened a new log
I20260810 22:56:23.282332 31200 ts_tablet_manager.cc:1397] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 22:56:23.283407 31202 log.cc:824] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:23.283452 31201 tablet_bootstrap.cc:654] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:23.285382 31201 log.cc:824] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:23.286175 31202 tablet_bootstrap.cc:492] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7: No bootstrap required, opened a new log
I20260810 22:56:23.286410 31202 ts_tablet_manager.cc:1397] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260810 22:56:23.287230 31200 raft_consensus.cc:348] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.287539 31200 raft_consensus.cc:374] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:23.287603 31200 raft_consensus.cc:729] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec2abf5597c040219a123809c132ddce, State: Initialized, Role: FOLLOWER
I20260810 22:56:23.287940 31201 tablet_bootstrap.cc:492] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d: No bootstrap required, opened a new log
I20260810 22:56:23.288169 31201 ts_tablet_manager.cc:1397] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20260810 22:56:23.288113 31200 consensus_queue.cc:260] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.288971 31027 heartbeater.cc:503] Master 127.29.78.190:40643 was elected leader, sending a full tablet report...
I20260810 22:56:23.289254 31200 ts_tablet_manager.cc:1428] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:56:23.292092 31202 raft_consensus.cc:348] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.292407 31202 raft_consensus.cc:374] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:23.292465 31202 raft_consensus.cc:729] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5282abfc11b7467099b22a3f53a1cdf7, State: Initialized, Role: FOLLOWER
I20260810 22:56:23.292961 31202 consensus_queue.cc:260] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.293699 30877 heartbeater.cc:503] Master 127.29.78.190:40643 was elected leader, sending a full tablet report...
I20260810 22:56:23.293747 31201 raft_consensus.cc:348] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.294066 31202 ts_tablet_manager.cc:1428] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260810 22:56:23.294102 31201 raft_consensus.cc:374] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:23.294157 31201 raft_consensus.cc:729] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65db365f572c4d4783856596f522815d, State: Initialized, Role: FOLLOWER
I20260810 22:56:23.294548 31201 consensus_queue.cc:260] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.295257 31180 heartbeater.cc:503] Master 127.29.78.190:40643 was elected leader, sending a full tablet report...
I20260810 22:56:23.295480 31201 ts_tablet_manager.cc:1428] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d: Time spent starting tablet: real 0.007s user 0.007s sys 0.002s
W20260810 22:56:23.459666 31181 tablet.cc:2369] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:23.490772 31208 raft_consensus.cc:482] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:23.490967 31208 raft_consensus.cc:504] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.492096 31208 leader_election.cc:288] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65db365f572c4d4783856596f522815d (127.29.78.131:45265), ec2abf5597c040219a123809c132ddce (127.29.78.130:41397)
I20260810 22:56:23.496778 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.492661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:50603 (local address 127.29.78.130:41397)
0810 22:56:23.492800 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.492804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.493239 (+ 435us) server_negotiation.cc:407] Connection header received
0810 22:56:23.493400 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.493403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.493458 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.493544 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:23.493916 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.494442 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.495737 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.496125 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.496294 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.496351 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.496592 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.496666 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:56:23.496837 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.492371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:41397 (local address 127.29.78.129:50603)
0810 22:56:23.493031 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:23.493043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:23.493279 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:23.493556 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:23.493558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:23.493584 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:23.493753 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.493759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.494543 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.494548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.495610 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.495619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.496312 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.496316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.496461 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.496480 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:23.496574 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:56:23.496643 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":140,"spinlock_wait_cycles":1280,"thread_start_us":172,"threads_started":1}
I20260810 22:56:23.497081 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.492547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:56839 (local address 127.29.78.131:45265)
0810 22:56:23.492731 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.492736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.493133 (+ 397us) server_negotiation.cc:407] Connection header received
0810 22:56:23.493300 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.493303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.493353 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.493437 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:23.493990 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.494476 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.495865 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.496446 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.496593 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.496662 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.496899 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.496968 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 22:56:23.497144 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.492370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:45265 (local address 127.29.78.129:56839)
0810 22:56:23.492994 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:23.493020 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:23.493177 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:23.493451 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:23.493454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:23.493470 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:23.493868 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.493883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.494586 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.494591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.495741 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:23.495751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.496585 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.496588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:23.496771 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.496790 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:23.496884 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:56:23.496954 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"spinlock_wait_cycles":5632,"thread_start_us":160,"threads_started":1}
I20260810 22:56:23.497920 31129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55695c08904e8f9aafd3054e497589" candidate_uuid: "5282abfc11b7467099b22a3f53a1cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65db365f572c4d4783856596f522815d" is_pre_election: true
I20260810 22:56:23.498095 30978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55695c08904e8f9aafd3054e497589" candidate_uuid: "5282abfc11b7467099b22a3f53a1cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2abf5597c040219a123809c132ddce" is_pre_election: true
I20260810 22:56:23.498450 31129 raft_consensus.cc:2454] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5282abfc11b7467099b22a3f53a1cdf7 in term 0.
I20260810 22:56:23.498987 30754 leader_election.cc:302] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5282abfc11b7467099b22a3f53a1cdf7, 65db365f572c4d4783856596f522815d; no voters:
I20260810 22:56:23.499234 31208 raft_consensus.cc:2781] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:23.499332 31208 raft_consensus.cc:482] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:23.499385 31208 raft_consensus.cc:3037] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 0 FOLLOWER]: Advancing to term 1
W20260810 22:56:23.499586 31028 tablet.cc:2369] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:23.500545 30978 raft_consensus.cc:2454] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5282abfc11b7467099b22a3f53a1cdf7 in term 0.
I20260810 22:56:23.501278 31208 raft_consensus.cc:504] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.501631 31208 leader_election.cc:288] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [CANDIDATE]: Term 1 election: Requested vote from peers 65db365f572c4d4783856596f522815d (127.29.78.131:45265), ec2abf5597c040219a123809c132ddce (127.29.78.130:41397)
I20260810 22:56:23.501979 30978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55695c08904e8f9aafd3054e497589" candidate_uuid: "5282abfc11b7467099b22a3f53a1cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec2abf5597c040219a123809c132ddce"
I20260810 22:56:23.502017 31129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b55695c08904e8f9aafd3054e497589" candidate_uuid: "5282abfc11b7467099b22a3f53a1cdf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65db365f572c4d4783856596f522815d"
I20260810 22:56:23.502099 30978 raft_consensus.cc:3037] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:23.502117 31129 raft_consensus.cc:3037] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:23.504122 31129 raft_consensus.cc:2454] T 8b55695c08904e8f9aafd3054e497589 P 65db365f572c4d4783856596f522815d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5282abfc11b7467099b22a3f53a1cdf7 in term 1.
I20260810 22:56:23.504122 30978 raft_consensus.cc:2454] T 8b55695c08904e8f9aafd3054e497589 P ec2abf5597c040219a123809c132ddce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5282abfc11b7467099b22a3f53a1cdf7 in term 1.
I20260810 22:56:23.504416 30757 leader_election.cc:302] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5282abfc11b7467099b22a3f53a1cdf7, ec2abf5597c040219a123809c132ddce; no voters:
I20260810 22:56:23.504575 31208 raft_consensus.cc:2781] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:23.504819 31208 raft_consensus.cc:686] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [term 1 LEADER]: Becoming Leader. State: Replica: 5282abfc11b7467099b22a3f53a1cdf7, State: Running, Role: LEADER
I20260810 22:56:23.505151 31208 consensus_queue.cc:237] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } }
I20260810 22:56:23.508375 30666 catalog_manager.cc:5495] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5282abfc11b7467099b22a3f53a1cdf7 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "5282abfc11b7467099b22a3f53a1cdf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5282abfc11b7467099b22a3f53a1cdf7" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65db365f572c4d4783856596f522815d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 45265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec2abf5597c040219a123809c132ddce" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 41397 } health_report { overall_health: UNKNOWN } } }
W20260810 22:56:23.524394 30878 tablet.cc:2369] T 8b55695c08904e8f9aafd3054e497589 P 5282abfc11b7467099b22a3f53a1cdf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:23.554879 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.545195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.29.78.129:46303)
0810 22:56:23.545342 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.545347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.545661 (+ 314us) server_negotiation.cc:407] Connection header received
0810 22:56:23.545845 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.545847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.545889 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.545968 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:23.546694 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.547142 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.548141 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.548331 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.548425 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.551217 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:23.551228 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:23.551240 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:23.551262 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:23.552905 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.553404 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.553410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.553411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.553491 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.553769 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.553773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.553775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.554084 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:23.554421 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.554570 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.554681 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402583, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:23.582017 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.573912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41422 (local address 127.29.78.129:46303)
0810 22:56:23.574045 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.574050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.574214 (+ 164us) server_negotiation.cc:407] Connection header received
0810 22:56:23.574365 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.574375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.574414 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.574496 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:23.575136 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.575610 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.576450 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.576670 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.576845 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.579130 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:23.579133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:23.579136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:23.579163 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:23.580346 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.580741 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.580745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.580747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.580823 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.581156 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.581159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.581160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.581326 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:23.581555 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.581726 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.581903 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[30614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:23.606580 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:23.598797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.29.78.129:46303)
0810 22:56:23.598991 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:23.598994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:23.599082 (+ 88us) server_negotiation.cc:407] Connection header received
0810 22:56:23.599184 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:23.599186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:23.599221 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:23.599284 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:23.599947 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.600457 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.601248 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:23.601430 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:23.601496 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:23.603835 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:23.603837 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:23.603839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:23.603860 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:23.605057 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.605422 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.605424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.605426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.605508 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:23.605752 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:23.605754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:23.605755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:23.605997 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:23.606104 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:23.606303 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 22:56:23.606387 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260810 22:56:23.608058 30784 tablet_service.cc:2908] Rejecting scan request for tablet 8b55695c08904e8f9aafd3054e497589: Uninitialized: clean time has not yet been initialized
I20260810 22:56:23.610258 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 30734
I20260810 22:56:23.621891 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 30884
I20260810 22:56:23.628167 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 31034
I20260810 22:56:23.634538 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 30628
2026-08-10T22:56:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2431 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[31232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[31232](info): set up 2 sockets
Aug 10 22:56:23 dist-test-slave-6nmv krb5kdc[31232](info): commencing operation
krb5kdc: starting...
W20260810 22:56:24.667214 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:24 dist-test-slave-6nmv krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:24.786906 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40345
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:24.902958 31248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:24.903453 31248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:24.903539 31248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:24.908461 31248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:24.908519 31248 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:56:24.908550 31248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:24.908567 31248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:24.908581 31248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:24.908595 31248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:24.913988 31248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40345
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:24.915079 31248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:24.916123 31248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:24.925418 31254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:24.925621 31248 server_base.cc:1034] running on GCE node
W20260810 22:56:24.925622 31256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:24.925441 31253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:24.926590 31248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:24.927631 31248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:24.928829 31248 hybrid_clock.cc:648] HybridClock initialized: now 1786402584928795 us; error 53 us; skew 500 ppm
Aug 10 22:56:24 dist-test-slave-6nmv krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402584, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:24.931764 31248 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:24.932693 31248 webserver.cc:460] Webserver started at http://127.29.78.190:33291/ using document root <none> and password file <none>
I20260810 22:56:24.933262 31248 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:24.933355 31248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:24.933595 31248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:24.935693 31248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "2ae681dbaed84c0faa359fa0de7c834f"
format_stamp: "Formatted at 2026-08-10 22:56:24 on dist-test-slave-6nmv"
server_key: "372b518ec0c0cd1c04f367765ae583ab"
server_key_iv: "13bfd8f966d1f90519b00c2d32c17461"
server_key_version: "encryptionkey@0"
I20260810 22:56:24.936272 31248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "2ae681dbaed84c0faa359fa0de7c834f"
format_stamp: "Formatted at 2026-08-10 22:56:24 on dist-test-slave-6nmv"
server_key: "372b518ec0c0cd1c04f367765ae583ab"
server_key_iv: "13bfd8f966d1f90519b00c2d32c17461"
server_key_version: "encryptionkey@0"
I20260810 22:56:24.939987 31248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:24.942487 31264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:24.943786 31248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:24.943930 31248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "2ae681dbaed84c0faa359fa0de7c834f"
format_stamp: "Formatted at 2026-08-10 22:56:24 on dist-test-slave-6nmv"
server_key: "372b518ec0c0cd1c04f367765ae583ab"
server_key_iv: "13bfd8f966d1f90519b00c2d32c17461"
server_key_version: "encryptionkey@0"
I20260810 22:56:24.944049 31248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:24.962368 31248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:24.965909 31248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:24.966125 31248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:24.974800 31248 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40345
I20260810 22:56:24.974817 31334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40345 every 8 connection(s)
I20260810 22:56:24.975795 31248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:24.979264 31335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:24.983568 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 31248
I20260810 22:56:24.983705 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:24.984025 30010 external_mini_cluster.cc:1428] Setting key 1d017ba4eaeae7362ed94d5c70cfa981
I20260810 22:56:24.986670 31335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f: Bootstrap starting.
I20260810 22:56:24.989089 31335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:24.989996 31335 log.cc:824] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:24.992519 31335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f: No bootstrap required, opened a new log
Aug 10 22:56:24 dist-test-slave-6nmv krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:24.996755 31335 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae681dbaed84c0faa359fa0de7c834f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40345 } }
I20260810 22:56:24.996992 31335 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:24.997067 31335 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae681dbaed84c0faa359fa0de7c834f, State: Initialized, Role: FOLLOWER
I20260810 22:56:24.997583 31335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae681dbaed84c0faa359fa0de7c834f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40345 } }
I20260810 22:56:24.997757 31335 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:24.997861 31335 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:24.997972 31335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:24.999749 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:24.985631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39810 (local address 127.29.78.190:40345)
0810 22:56:24.986329 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:24.986354 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:24.986395 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:56:24.987130 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:24.987148 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:24.987478 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:24.987847 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:24.988674 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:24.989725 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:24.990917 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:24.991496 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:24.991656 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:24.994817 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:24.994822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:24.994835 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:24.994869 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:24.996624 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:24.997143 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:24.997151 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:24.997159 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:24.997288 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:24.997640 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:24.997642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:24.997643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:24.998226 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:24.998375 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:24.998662 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 22:56:24.998999 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":178,"threads_started":1}
I20260810 22:56:25.040019 31335 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae681dbaed84c0faa359fa0de7c834f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40345 } }
I20260810 22:56:25.040560 31335 leader_election.cc:302] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ae681dbaed84c0faa359fa0de7c834f; no voters:
I20260810 22:56:25.040992 31335 leader_election.cc:288] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:25.041149 31342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:25.041445 31342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [term 1 LEADER]: Becoming Leader. State: Replica: 2ae681dbaed84c0faa359fa0de7c834f, State: Running, Role: LEADER
I20260810 22:56:25.041882 31342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae681dbaed84c0faa359fa0de7c834f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40345 } }
I20260810 22:56:25.042543 31335 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:25.043691 31344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ae681dbaed84c0faa359fa0de7c834f. Latest consensus state: current_term: 1 leader_uuid: "2ae681dbaed84c0faa359fa0de7c834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae681dbaed84c0faa359fa0de7c834f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40345 } } }
I20260810 22:56:25.043851 31344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:25.044202 31343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ae681dbaed84c0faa359fa0de7c834f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ae681dbaed84c0faa359fa0de7c834f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40345 } } }
I20260810 22:56:25.044342 31343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:25.044802 31352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:25.048727 31352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:25.054375 31352 catalog_manager.cc:1347] Generated new cluster ID: dd6a141a51e04105bc2770caeea5ef46
I20260810 22:56:25.054474 31352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:25.075487 31352 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:25.076817 31352 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:25.082036 31352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f: Generated new TSK 0
I20260810 22:56:25.082813 31352 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
W20260810 22:56:25.832310 30010 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.78.129: real 0.745s user 0.000s sys 0.004s
I20260810 22:56:25.832880 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40345
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:25.950815 31367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:25.951124 31367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:25.951223 31367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:25.955545 31367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:25.955634 31367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:25.955739 31367 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:25.955826 31367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:25.961227 31367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:40345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:25.962316 31367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:25.963442 31367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:25.973295 31376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:25.973286 31373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:25.973291 31372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:25.973531 31367 server_base.cc:1034] running on GCE node
I20260810 22:56:25.974128 31367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:25.974798 31367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:25.975986 31367 hybrid_clock.cc:648] HybridClock initialized: now 1786402585975974 us; error 30 us; skew 500 ppm
Aug 10 22:56:25 dist-test-slave-6nmv krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402585, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:25.979200 31367 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:25.980185 31367 webserver.cc:460] Webserver started at http://127.29.78.129:33111/ using document root <none> and password file <none>
I20260810 22:56:25.980720 31367 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:25.980779 31367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:25.980984 31367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:25.983086 31367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "c94538c4842140b49cb9ec5e6b4d31a9"
format_stamp: "Formatted at 2026-08-10 22:56:25 on dist-test-slave-6nmv"
server_key: "466506bdafb3d6907aada52b1e9127cf"
server_key_iv: "1409ee7dfb09d8f9db82c69d3b46be69"
server_key_version: "encryptionkey@0"
I20260810 22:56:25.983675 31367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "c94538c4842140b49cb9ec5e6b4d31a9"
format_stamp: "Formatted at 2026-08-10 22:56:25 on dist-test-slave-6nmv"
server_key: "466506bdafb3d6907aada52b1e9127cf"
server_key_iv: "1409ee7dfb09d8f9db82c69d3b46be69"
server_key_version: "encryptionkey@0"
I20260810 22:56:25.987362 31367 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 22:56:25.990149 31385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:25.991364 31367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:56:25.991499 31367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "c94538c4842140b49cb9ec5e6b4d31a9"
format_stamp: "Formatted at 2026-08-10 22:56:25 on dist-test-slave-6nmv"
server_key: "466506bdafb3d6907aada52b1e9127cf"
server_key_iv: "1409ee7dfb09d8f9db82c69d3b46be69"
server_key_version: "encryptionkey@0"
I20260810 22:56:25.991627 31367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:26.003342 31367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:26.006724 31367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:26.006919 31367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:26.008327 31367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:26.008401 31367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:26.008471 31367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:26.008519 31367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:26 dist-test-slave-6nmv krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402585, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:26.020046 31367 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:34887
I20260810 22:56:26.020109 31516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:34887 every 8 connection(s)
I20260810 22:56:26.021029 31367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:26.023258 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.009298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46573 (local address 127.29.78.190:40345)
0810 22:56:26.009570 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:26.009575 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:26.010419 (+ 844us) server_negotiation.cc:407] Connection header received
0810 22:56:26.011701 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:26.011707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:26.011780 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:26.011903 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:26.013233 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.013993 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.015141 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.015382 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.015500 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.019276 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:26.019279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:26.019281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:26.019305 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:26.020992 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.021594 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.021598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.021600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.021689 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.022126 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.022129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.022131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.022385 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:26.022485 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:26.022988 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 22:56:26.023125 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260810 22:56:26.023972 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.009684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40345 (local address 127.29.78.129:46573)
0810 22:56:26.010233 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:26.010294 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:26.011424 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:26.012079 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:26.012086 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:26.012470 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:26.013059 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.013096 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.014128 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.014131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.015009 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.015018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.015548 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.015553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.015746 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.016408 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:26.016424 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:26.019101 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:26.021117 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.021123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.021141 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.021486 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.021789 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.021795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.021800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.022004 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.022493 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:26.022503 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:26.022731 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 22:56:26.022877 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 22:56:26.023149 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":161,"threads_started":1}
I20260810 22:56:26.025169 31517 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40345
I20260810 22:56:26.025324 31517 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:26.025882 31517 heartbeater.cc:511] Master 127.29.78.190:40345 requested a full tablet report, sending...
I20260810 22:56:26.027460 31283 ts_manager.cc:194] Registered new tserver with Master: c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129:34887)
I20260810 22:56:26.028594 31283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:46573
I20260810 22:56:26.029306 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 31367
I20260810 22:56:26.029433 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:26.029691 30010 external_mini_cluster.cc:1428] Setting key 6c4f2c978599fcba50878f0134bb0de5
I20260810 22:56:26.034989 31395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.027616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:38353 (local address 127.29.78.190:40345)
0810 22:56:26.027754 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:26.027758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:26.027946 (+ 188us) server_negotiation.cc:407] Connection header received
0810 22:56:26.028123 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:26.028126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:26.028168 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:26.028245 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:26.028657 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.029209 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.030374 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.030709 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.030787 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.031920 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:26.031924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:26.031926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:26.031949 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:26.033345 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.033781 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.033784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.033786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.033897 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.034182 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.034186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.034188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.034354 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:26.034479 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:26.034738 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 22:56:26.034849 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:56:26.034997 31523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.027550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40345 (local address 127.29.78.129:38353)
0810 22:56:26.027864 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:26.027876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:26.027984 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:26.028262 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:26.028265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:26.028452 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:26.028544 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.028549 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.029344 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.029349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.030240 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.030249 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.030929 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.030931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.031000 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.031451 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:26.031462 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:26.031787 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:26.033456 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.033461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.033462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.033671 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.033997 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.034000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.034002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.034089 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.034471 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:26.034474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:26.034571 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 22:56:26.034671 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 22:56:26.034817 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
W20260810 22:56:26.715283 30010 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.29.78.130: real 0.684s user 0.004s sys 0.000s
I20260810 22:56:26.715806 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40345
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:26.825906 31526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:26.826167 31526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:26.826210 31526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:26.829995 31526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:26.830044 31526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:26.830082 31526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:26.830112 31526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:26.835254 31526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:40345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:26.836140 31526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:26.837070 31526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:26.846169 31536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:26.846238 31534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:26.846169 31533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:26.846323 31526 server_base.cc:1034] running on GCE node
I20260810 22:56:26.846776 31526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:26.847400 31526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:26.848594 31526 hybrid_clock.cc:648] HybridClock initialized: now 1786402586848583 us; error 24 us; skew 500 ppm
Aug 10 22:56:26 dist-test-slave-6nmv krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402586, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:26.851234 31526 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:26.852101 31526 webserver.cc:460] Webserver started at http://127.29.78.130:39871/ using document root <none> and password file <none>
I20260810 22:56:26.852625 31526 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:26.852710 31526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:26.852941 31526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:26.855000 31526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "c41aecfa088e42c3b892498d2be52d8f"
format_stamp: "Formatted at 2026-08-10 22:56:26 on dist-test-slave-6nmv"
server_key: "6539eec755618a90602657b5526aa9c5"
server_key_iv: "f5de32453bdf30215d27d4013b8cb4b3"
server_key_version: "encryptionkey@0"
I20260810 22:56:26.855548 31526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "c41aecfa088e42c3b892498d2be52d8f"
format_stamp: "Formatted at 2026-08-10 22:56:26 on dist-test-slave-6nmv"
server_key: "6539eec755618a90602657b5526aa9c5"
server_key_iv: "f5de32453bdf30215d27d4013b8cb4b3"
server_key_version: "encryptionkey@0"
I20260810 22:56:26.859076 31526 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:56:26.861332 31546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:26.862583 31526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:56:26.862689 31526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "c41aecfa088e42c3b892498d2be52d8f"
format_stamp: "Formatted at 2026-08-10 22:56:26 on dist-test-slave-6nmv"
server_key: "6539eec755618a90602657b5526aa9c5"
server_key_iv: "f5de32453bdf30215d27d4013b8cb4b3"
server_key_version: "encryptionkey@0"
I20260810 22:56:26.862780 31526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:26.907215 31526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:26.910152 31526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:26.910300 31526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:26.911674 31526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:26.911758 31526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:26.911827 31526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:26.911881 31526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:26 dist-test-slave-6nmv krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402586, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:26.922646 31526 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:33425
I20260810 22:56:26.922748 31673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:33425 every 8 connection(s)
I20260810 22:56:26.923578 31526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:26.925632 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.912689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44379 (local address 127.29.78.190:40345)
0810 22:56:26.912961 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:26.912968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:26.913895 (+ 927us) server_negotiation.cc:407] Connection header received
0810 22:56:26.915017 (+ 1122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:26.915021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:26.915113 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:26.915189 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:26.916612 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.917453 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.918641 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.918858 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.918991 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.921862 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:26.921866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:26.921869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:26.921896 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:26.923471 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.924068 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.924072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.924074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.924169 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.924612 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.924615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.924617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.924787 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:26.924886 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:26.925370 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 22:56:26.925469 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":102,"threads_started":1}
I20260810 22:56:26.926424 31558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.913217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40345 (local address 127.29.78.130:44379)
0810 22:56:26.913725 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:26.913766 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:26.914753 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:26.915631 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:26.915637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:26.915928 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:26.916460 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.916483 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.917587 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.917590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.918513 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.918523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.919042 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.919045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.919239 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.919812 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:26.919831 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:26.921653 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:26.923586 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.923595 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.923609 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.923941 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.924281 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.924284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.924286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.924486 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.924938 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:26.924943 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:26.925188 (+ 245us) client_negotiation.cc:769] Sending connection context
0810 22:56:26.925328 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 22:56:26.925616 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":137,"threads_started":1}
I20260810 22:56:26.927619 31675 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40345
I20260810 22:56:26.927820 31675 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:26.928298 31675 heartbeater.cc:511] Master 127.29.78.190:40345 requested a full tablet report, sending...
I20260810 22:56:26.929283 31284 ts_manager.cc:194] Registered new tserver with Master: c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425)
I20260810 22:56:26.930020 31284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:44379
I20260810 22:56:26.933399 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 31526
I20260810 22:56:26.933547 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:26.933935 30010 external_mini_cluster.cc:1428] Setting key 4f13c4ed7f4ba0ba4a0c7d9f784083ef
I20260810 22:56:26.936795 31682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.929754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40345 (local address 127.29.78.130:51375)
0810 22:56:26.930057 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:26.930069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:26.930164 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:26.930516 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:26.930521 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:26.930711 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:26.930799 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.930804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.931755 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.931757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.932304 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:26.932311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.932719 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.932721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:26.932789 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.933235 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:26.933249 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:26.933596 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:26.935361 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.935363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.935364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.935542 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.935832 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:26.935834 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:26.935835 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:26.935905 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:26.936422 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:26.936424 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:26.936494 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 22:56:26.936577 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 22:56:26.936674 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":137,"threads_started":1}
I20260810 22:56:26.937027 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:26.929826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:51375 (local address 127.29.78.190:40345)
0810 22:56:26.930331 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:26.930335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:26.930349 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:26.930388 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:26.930389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:26.930420 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:26.930475 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:26.930910 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.931602 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.932404 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:26.932623 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:26.932702 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:26.933735 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:26.933740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:26.933742 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:26.933769 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:26.935275 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.935641 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.935647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.935649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.935737 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:26.936021 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:26.936027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:26.936031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:26.936326 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:26.936433 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:26.936659 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 22:56:26.936793 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:26.962218 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40345
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:56:27.031008 31517 heartbeater.cc:503] Master 127.29.78.190:40345 was elected leader, sending a full tablet report...
W20260810 22:56:27.072084 31685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:27.072319 31685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:27.072366 31685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:27.076279 31685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:27.076332 31685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:27.076391 31685 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:27.076434 31685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:27.081226 31685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:40345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:27.082221 31685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:27.083225 31685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:27.092478 31691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:27.092625 31685 server_base.cc:1034] running on GCE node
W20260810 22:56:27.092512 31694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:27.092823 31690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:27.093277 31685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:27.093990 31685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:27.095152 31685 hybrid_clock.cc:648] HybridClock initialized: now 1786402587095132 us; error 32 us; skew 500 ppm
Aug 10 22:56:27 dist-test-slave-6nmv krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402587, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:27.097965 31685 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:27.098814 31685 webserver.cc:460] Webserver started at http://127.29.78.131:37111/ using document root <none> and password file <none>
I20260810 22:56:27.099359 31685 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:27.099447 31685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:27.099668 31685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:27.101553 31685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "3000506ef956460eb065859b1b034ac6"
format_stamp: "Formatted at 2026-08-10 22:56:27 on dist-test-slave-6nmv"
server_key: "30dac3072e3323a98d030844c93a2c39"
server_key_iv: "35ccf7d1b4f7a738785bb734bf844eff"
server_key_version: "encryptionkey@0"
I20260810 22:56:27.102095 31685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "3000506ef956460eb065859b1b034ac6"
format_stamp: "Formatted at 2026-08-10 22:56:27 on dist-test-slave-6nmv"
server_key: "30dac3072e3323a98d030844c93a2c39"
server_key_iv: "35ccf7d1b4f7a738785bb734bf844eff"
server_key_version: "encryptionkey@0"
I20260810 22:56:27.105573 31685 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:56:27.107849 31702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:27.108933 31685 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:27.109064 31685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "3000506ef956460eb065859b1b034ac6"
format_stamp: "Formatted at 2026-08-10 22:56:27 on dist-test-slave-6nmv"
server_key: "30dac3072e3323a98d030844c93a2c39"
server_key_iv: "35ccf7d1b4f7a738785bb734bf844eff"
server_key_version: "encryptionkey@0"
I20260810 22:56:27.109159 31685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:27.121713 31685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:27.124879 31685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:27.125125 31685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:27.126688 31685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:27.126736 31685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:27.126793 31685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:27.126859 31685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:27 dist-test-slave-6nmv krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402587, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:27.137799 31685 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:42755
I20260810 22:56:27.137887 31832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:42755 every 8 connection(s)
I20260810 22:56:27.138832 31685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:27.140148 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.127508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:57873 (local address 127.29.78.190:40345)
0810 22:56:27.127666 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.127672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.129248 (+ 1576us) server_negotiation.cc:407] Connection header received
0810 22:56:27.130430 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.130434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.130494 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.130593 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:27.131670 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.132198 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.133359 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.133676 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.133795 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.136359 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:27.136362 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:27.136364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:27.136388 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:27.138178 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.138683 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.138687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.138688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.138790 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.139159 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.139164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.139166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.139389 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:27.139488 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.139947 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.140043 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:56:27.140803 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.128566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40345 (local address 127.29.78.131:57873)
0810 22:56:27.129079 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.129131 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.130223 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.130760 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.130766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.131082 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:27.131507 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.131519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.132325 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.132328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.133212 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.133222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.133868 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.133873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.134052 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.134631 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:27.134647 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:27.136188 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:27.138313 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:27.138319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:27.138337 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:27.138581 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:27.138932 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:27.138935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:27.138937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:27.139063 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:27.139554 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:27.139559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:27.139757 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.139889 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.140092 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":125,"threads_started":1}
I20260810 22:56:27.141955 31833 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40345
I20260810 22:56:27.142118 31833 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:27.142544 31833 heartbeater.cc:511] Master 127.29.78.190:40345 requested a full tablet report, sending...
I20260810 22:56:27.143594 31283 ts_manager.cc:194] Registered new tserver with Master: 3000506ef956460eb065859b1b034ac6 (127.29.78.131:42755)
I20260810 22:56:27.144388 31283 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:57873
I20260810 22:56:27.148649 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 31685
I20260810 22:56:27.148784 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:27.149119 30010 external_mini_cluster.cc:1428] Setting key 1af0e92d04190983a729226ee3100613
I20260810 22:56:27.151018 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.144071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:44077 (local address 127.29.78.190:40345)
0810 22:56:27.144180 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.144184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.144348 (+ 164us) server_negotiation.cc:407] Connection header received
0810 22:56:27.144486 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.144489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.144543 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.144614 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:27.145060 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.145539 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.146498 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.146681 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.146752 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.147678 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:27.147687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:27.147689 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:27.147710 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:27.149366 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.149866 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.149870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.149874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.149988 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.150265 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.150270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.150272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.150438 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:27.150556 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.150787 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.150882 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 22:56:27.151021 31839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.144007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40345 (local address 127.29.78.131:44077)
0810 22:56:27.144280 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.144295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.144389 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.144661 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.144665 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.144837 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:27.144955 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.144962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.145672 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.145675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.146403 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.146412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.146768 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.146770 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.146830 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.147233 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:27.147254 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:27.147532 (+ 278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:27.149487 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:27.149492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:27.149493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:27.149741 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:27.150080 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:27.150084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:27.150086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:27.150170 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:27.150551 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:27.150553 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:27.150640 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.150738 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.150865 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":117,"threads_started":1}
I20260810 22:56:27.151216 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:56:27 dist-test-slave-6nmv krb5kdc[31232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:56:27 dist-test-slave-6nmv krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:27.177403 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.169187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.29.78.190:40345)
0810 22:56:27.169347 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.169353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.169370 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:56:27.169540 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.169543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.169585 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.169665 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:27.170381 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.170857 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.171680 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.171919 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.172036 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.174360 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:27.174363 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:27.174368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:27.174398 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:27.175746 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.176160 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.176165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.176167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.176250 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.176560 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.176563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.176565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.176742 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:27.176887 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.177121 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.177238 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:56:27.180228 31283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:27.183305 31283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:27.197242 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.191358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.29.78.131:42755)
0810 22:56:27.192235 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.192249 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.192275 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:56:27.192347 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.192352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.192550 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.192724 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:27.192987 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.194179 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.195903 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.196559 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.196671 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.196826 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.197014 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.197097 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":150,"threads_started":1}
I20260810 22:56:27.197230 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.191212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42755 (local address 127.0.0.1:58086)
0810 22:56:27.191735 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.191802 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.191928 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.192713 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.192719 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.192741 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:27.192881 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.192887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.194296 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.194301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.195784 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.195793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.196673 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.196676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.196792 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.196833 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.196979 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.197092 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":91,"thread_start_us":93,"threads_started":1}
I20260810 22:56:27.197659 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.191975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:33425 (local address 127.0.0.1:36560)
0810 22:56:27.193246 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.193259 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.193372 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.193795 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.193798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.193838 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:27.193931 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.193936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.195257 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.195260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.196305 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.196315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.197273 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.197277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.197367 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.197390 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.197475 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.197542 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1188,"thread_start_us":85,"threads_started":1}
I20260810 22:56:27.197715 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.192135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.29.78.130:33425)
0810 22:56:27.192516 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.192527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.193378 (+ 851us) server_negotiation.cc:407] Connection header received
0810 22:56:27.193462 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.193467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.193648 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.193792 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:27.194043 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.195139 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.196440 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.197154 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.197271 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.197410 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.197508 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.197589 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":102,"threads_started":1}
I20260810 22:56:27.198465 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.191387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34887 (local address 127.0.0.1:56612)
0810 22:56:27.192259 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.192274 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.192363 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.193025 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.193029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.193049 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:27.193123 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.193128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.194384 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.194387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.195610 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.195619 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.196688 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.196692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.198141 (+ 1449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.198164 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.198241 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.198309 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"mutex_wait_us":1322,"thread_start_us":83,"threads_started":1}
I20260810 22:56:27.200384 31760 tablet_service.cc:1467] Processing CreateTablet for tablet caf6041962b44afea8eaf3041cee7b97 (DEFAULT_TABLE table=test-table [id=45bb94ece018456c94681cb8cae19234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:27.200384 31603 tablet_service.cc:1467] Processing CreateTablet for tablet caf6041962b44afea8eaf3041cee7b97 (DEFAULT_TABLE table=test-table [id=45bb94ece018456c94681cb8cae19234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:27.201256 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.192068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.29.78.129:34887)
0810 22:56:27.192621 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.192628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.192650 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:56:27.192724 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.192731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.192924 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.193043 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:27.193243 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.194208 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.195749 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.196564 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.196788 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.196934 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.201004 (+ 4070us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.201113 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":84,"threads_started":1}
I20260810 22:56:27.201463 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf6041962b44afea8eaf3041cee7b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.201462 31603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf6041962b44afea8eaf3041cee7b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.203418 31443 tablet_service.cc:1467] Processing CreateTablet for tablet caf6041962b44afea8eaf3041cee7b97 (DEFAULT_TABLE table=test-table [id=45bb94ece018456c94681cb8cae19234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:27.204284 31443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caf6041962b44afea8eaf3041cee7b97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.210131 31860 tablet_bootstrap.cc:492] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Bootstrap starting.
I20260810 22:56:27.210337 31858 tablet_bootstrap.cc:492] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Bootstrap starting.
I20260810 22:56:27.210515 31859 tablet_bootstrap.cc:492] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Bootstrap starting.
I20260810 22:56:27.212157 31860 tablet_bootstrap.cc:654] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.212397 31858 tablet_bootstrap.cc:654] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.212416 31859 tablet_bootstrap.cc:654] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.212911 31860 log.cc:824] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:27.213171 31858 log.cc:824] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:27.213171 31859 log.cc:824] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:27.214788 31860 tablet_bootstrap.cc:492] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: No bootstrap required, opened a new log
I20260810 22:56:27.214986 31860 ts_tablet_manager.cc:1397] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260810 22:56:27.215142 31859 tablet_bootstrap.cc:492] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: No bootstrap required, opened a new log
I20260810 22:56:27.215293 31859 ts_tablet_manager.cc:1397] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 22:56:27.215951 31858 tablet_bootstrap.cc:492] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: No bootstrap required, opened a new log
I20260810 22:56:27.216132 31858 ts_tablet_manager.cc:1397] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:56:27.218147 31860 raft_consensus.cc:348] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.218364 31860 raft_consensus.cc:374] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.218411 31860 raft_consensus.cc:729] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94538c4842140b49cb9ec5e6b4d31a9, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.218801 31860 consensus_queue.cc:260] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.218806 31859 raft_consensus.cc:348] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.219035 31859 raft_consensus.cc:374] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.219084 31859 raft_consensus.cc:729] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41aecfa088e42c3b892498d2be52d8f, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.219447 31859 consensus_queue.cc:260] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.219669 31860 ts_tablet_manager.cc:1428] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260810 22:56:27.219636 31858 raft_consensus.cc:348] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.219861 31858 raft_consensus.cc:374] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.219924 31858 raft_consensus.cc:729] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3000506ef956460eb065859b1b034ac6, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.220187 31675 heartbeater.cc:503] Master 127.29.78.190:40345 was elected leader, sending a full tablet report...
I20260810 22:56:27.220355 31859 ts_tablet_manager.cc:1428] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 22:56:27.220295 31858 consensus_queue.cc:260] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.221161 31858 ts_tablet_manager.cc:1428] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:56:27.221472 31833 heartbeater.cc:503] Master 127.29.78.190:40345 was elected leader, sending a full tablet report...
W20260810 22:56:27.273458 31518 tablet.cc:2369] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:27.347810 31866 raft_consensus.cc:482] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:27.348030 31866 raft_consensus.cc:504] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.349114 31866 leader_election.cc:288] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129:34887), c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425)
I20260810 22:56:27.353240 31712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.349395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34887 (local address 127.29.78.131:46497)
0810 22:56:27.349521 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.349539 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.349640 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.350070 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.350073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.350094 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:27.350216 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.350226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.351083 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.351086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.352068 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.352077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.352896 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.352898 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.352990 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.353030 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.353087 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.353141 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 22:56:27.353350 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.349516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46497 (local address 127.29.78.129:34887)
0810 22:56:27.349689 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.349695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.349713 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:56:27.349773 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.349776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.349887 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.350009 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:27.350346 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.350953 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.352199 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.352749 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.352894 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.352948 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.353129 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.353207 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 22:56:27.354126 31867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.349395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:33425 (local address 127.29.78.131:35391)
0810 22:56:27.350198 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.350212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.350314 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.350557 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.350559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.350583 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:27.351171 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.351180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.351922 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.351926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.352729 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.352737 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.353651 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.353656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.353752 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.353774 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.353915 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.353984 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":232,"thread_start_us":332,"threads_started":1}
I20260810 22:56:27.354122 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.349549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:35391 (local address 127.29.78.130:33425)
0810 22:56:27.349715 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.349721 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.350314 (+ 593us) server_negotiation.cc:407] Connection header received
0810 22:56:27.350402 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.350405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.350463 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.350542 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:27.351306 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.351806 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.352853 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.353432 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.353553 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.353610 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.353938 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.354017 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:56:27.354176 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf6041962b44afea8eaf3041cee7b97" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" is_pre_election: true
I20260810 22:56:27.354542 31465 raft_consensus.cc:2454] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 0.
I20260810 22:56:27.354792 31625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf6041962b44afea8eaf3041cee7b97" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41aecfa088e42c3b892498d2be52d8f" is_pre_election: true
I20260810 22:56:27.355086 31625 raft_consensus.cc:2454] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 0.
I20260810 22:56:27.355068 31706 leader_election.cc:302] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3000506ef956460eb065859b1b034ac6, c94538c4842140b49cb9ec5e6b4d31a9; no voters:
I20260810 22:56:27.355363 31866 raft_consensus.cc:2781] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:27.355461 31866 raft_consensus.cc:482] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:27.355520 31866 raft_consensus.cc:3037] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.356581 31866 raft_consensus.cc:504] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.356949 31866 leader_election.cc:288] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 election: Requested vote from peers c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129:34887), c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425)
I20260810 22:56:27.357280 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf6041962b44afea8eaf3041cee7b97" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94538c4842140b49cb9ec5e6b4d31a9"
I20260810 22:56:27.357362 31625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caf6041962b44afea8eaf3041cee7b97" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41aecfa088e42c3b892498d2be52d8f"
I20260810 22:56:27.357431 31465 raft_consensus.cc:3037] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.357487 31625 raft_consensus.cc:3037] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.358668 31625 raft_consensus.cc:2454] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 1.
I20260810 22:56:27.358685 31465 raft_consensus.cc:2454] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 1.
I20260810 22:56:27.358963 31707 leader_election.cc:302] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3000506ef956460eb065859b1b034ac6, c41aecfa088e42c3b892498d2be52d8f; no voters:
I20260810 22:56:27.359157 31866 raft_consensus.cc:2781] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:27.359381 31866 raft_consensus.cc:686] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 1 LEADER]: Becoming Leader. State: Replica: 3000506ef956460eb065859b1b034ac6, State: Running, Role: LEADER
I20260810 22:56:27.359704 31866 consensus_queue.cc:237] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } }
I20260810 22:56:27.362797 31283 catalog_manager.cc:5495] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3000506ef956460eb065859b1b034ac6 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "3000506ef956460eb065859b1b034ac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } health_report { overall_health: UNKNOWN } } }
W20260810 22:56:27.389781 31834 tablet.cc:2369] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:27.396723 31854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.392567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.29.78.131:42755)
0810 22:56:27.392724 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.392729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.392743 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:27.392957 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.392965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.393020 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.393122 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:27.393417 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.394016 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.394961 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.395274 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.395414 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.395761 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:27.396322 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:27.396448 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.396542 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.396596 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:56:27.402688 31465 raft_consensus.cc:1261] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Refusing update from remote peer 3000506ef956460eb065859b1b034ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:27.403384 31866 consensus_queue.cc:1035] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:27.404866 31625 raft_consensus.cc:1261] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Refusing update from remote peer 3000506ef956460eb065859b1b034ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:27.405493 31866 consensus_queue.cc:1035] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:27.413025 31873 mvcc.cc:205] Tried to move back new op lower bound from 7317104997993385984 to 7317104997827776512. Current Snapshot: MvccSnapshot[applied={T|T < 7317104997993385984}]
I20260810 22:56:27.413699 31876 mvcc.cc:205] Tried to move back new op lower bound from 7317104997993385984 to 7317104997827776512. Current Snapshot: MvccSnapshot[applied={T|T < 7317104997993385984}]
I20260810 22:56:27.425201 31283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:27.425441 31283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:27.428056 31283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f: Sending DeleteTablet for 3 replicas of tablet caf6041962b44afea8eaf3041cee7b97
I20260810 22:56:27.428987 31760 tablet_service.cc:1514] Processing DeleteTablet for tablet caf6041962b44afea8eaf3041cee7b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:27 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:58086
I20260810 22:56:27.429023 31443 tablet_service.cc:1514] Processing DeleteTablet for tablet caf6041962b44afea8eaf3041cee7b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:27 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56612
I20260810 22:56:27.429589 31603 tablet_service.cc:1514] Processing DeleteTablet for tablet caf6041962b44afea8eaf3041cee7b97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:27 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36560
I20260810 22:56:27.429760 31884 tablet_replica.cc:330] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: stopping tablet replica
I20260810 22:56:27.430037 31886 tablet_replica.cc:330] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: stopping tablet replica
I20260810 22:56:27.430099 31884 raft_consensus.cc:2229] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:27.430286 31886 raft_consensus.cc:2229] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:27.430487 31884 raft_consensus.cc:2258] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:27.430555 31886 raft_consensus.cc:2258] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:27.430727 31283 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:27.431111 31283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:27.431881 31884 ts_tablet_manager.cc:1905] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:27.432158 31886 ts_tablet_manager.cc:1905] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:27.435245 31886 ts_tablet_manager.cc:1918] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:27.435348 31886 log.cc:1196] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/caf6041962b44afea8eaf3041cee7b97
I20260810 22:56:27.435681 31886 ts_tablet_manager.cc:1939] T caf6041962b44afea8eaf3041cee7b97 P c41aecfa088e42c3b892498d2be52d8f: Deleting consensus metadata
I20260810 22:56:27.435905 31884 ts_tablet_manager.cc:1918] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:27.436002 31884 log.cc:1196] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/caf6041962b44afea8eaf3041cee7b97
I20260810 22:56:27.436589 31443 tablet_service.cc:1467] Processing CreateTablet for tablet ba153b8da8934c569a002542a0c8c613 (DEFAULT_TABLE table=test-table [id=4e84f1ffd8024b9ca59c9537cf4c0332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:27.436928 31443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba153b8da8934c569a002542a0c8c613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.437016 31603 tablet_service.cc:1467] Processing CreateTablet for tablet ba153b8da8934c569a002542a0c8c613 (DEFAULT_TABLE table=test-table [id=4e84f1ffd8024b9ca59c9537cf4c0332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:27.437050 31270 catalog_manager.cc:4841] TS c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425): tablet caf6041962b44afea8eaf3041cee7b97 (table test-table [id=45bb94ece018456c94681cb8cae19234]) successfully deleted
I20260810 22:56:27.437281 31603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba153b8da8934c569a002542a0c8c613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.437431 31760 tablet_service.cc:1467] Processing CreateTablet for tablet ba153b8da8934c569a002542a0c8c613 (DEFAULT_TABLE table=test-table [id=4e84f1ffd8024b9ca59c9537cf4c0332]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:27.437660 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba153b8da8934c569a002542a0c8c613. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.440011 31859 tablet_bootstrap.cc:492] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: Bootstrap starting.
I20260810 22:56:27.440351 31858 tablet_bootstrap.cc:492] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: Bootstrap starting.
I20260810 22:56:27.441298 31859 tablet_bootstrap.cc:654] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.442067 31858 tablet_bootstrap.cc:654] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.442328 31885 tablet_replica.cc:330] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: stopping tablet replica
I20260810 22:56:27.443140 31860 tablet_bootstrap.cc:492] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: Bootstrap starting.
I20260810 22:56:27.443217 31859 tablet_bootstrap.cc:492] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: No bootstrap required, opened a new log
I20260810 22:56:27.443306 31859 ts_tablet_manager.cc:1397] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 22:56:27.443925 31859 raft_consensus.cc:348] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.444051 31859 raft_consensus.cc:374] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.444092 31859 raft_consensus.cc:729] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41aecfa088e42c3b892498d2be52d8f, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.444198 31859 consensus_queue.cc:260] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.444430 31859 ts_tablet_manager.cc:1428] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:56:27.444672 31885 raft_consensus.cc:2229] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:27.445039 31884 ts_tablet_manager.cc:1939] T caf6041962b44afea8eaf3041cee7b97 P 3000506ef956460eb065859b1b034ac6: Deleting consensus metadata
I20260810 22:56:27.446046 31270 catalog_manager.cc:4841] TS 3000506ef956460eb065859b1b034ac6 (127.29.78.131:42755): tablet caf6041962b44afea8eaf3041cee7b97 (table test-table [id=45bb94ece018456c94681cb8cae19234]) successfully deleted
I20260810 22:56:27.446417 31885 raft_consensus.cc:2258] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:27.444845 31860 tablet_bootstrap.cc:654] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.447922 31858 tablet_bootstrap.cc:492] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: No bootstrap required, opened a new log
I20260810 22:56:27.447999 31858 ts_tablet_manager.cc:1397] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260810 22:56:27.448354 31885 ts_tablet_manager.cc:1905] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:27.448717 31858 raft_consensus.cc:348] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.448813 31858 raft_consensus.cc:374] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.448840 31858 raft_consensus.cc:729] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3000506ef956460eb065859b1b034ac6, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.448944 31858 consensus_queue.cc:260] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.449165 31858 ts_tablet_manager.cc:1428] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:56:27.450457 31860 tablet_bootstrap.cc:492] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: No bootstrap required, opened a new log
I20260810 22:56:27.450744 31860 ts_tablet_manager.cc:1397] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260810 22:56:27.451296 31885 ts_tablet_manager.cc:1918] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:27.451397 31885 log.cc:1196] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/caf6041962b44afea8eaf3041cee7b97
I20260810 22:56:27.451699 31860 raft_consensus.cc:348] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.451797 31885 ts_tablet_manager.cc:1939] T caf6041962b44afea8eaf3041cee7b97 P c94538c4842140b49cb9ec5e6b4d31a9: Deleting consensus metadata
I20260810 22:56:27.451891 31860 raft_consensus.cc:374] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.452018 31860 raft_consensus.cc:729] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94538c4842140b49cb9ec5e6b4d31a9, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.452193 31860 consensus_queue.cc:260] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.452831 31860 ts_tablet_manager.cc:1428] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 22:56:27.453097 31268 catalog_manager.cc:4841] TS c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129:34887): tablet caf6041962b44afea8eaf3041cee7b97 (table test-table [id=45bb94ece018456c94681cb8cae19234]) successfully deleted
I20260810 22:56:27.557741 31880 raft_consensus.cc:482] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:27.557986 31880 raft_consensus.cc:504] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.558903 31880 leader_election.cc:288] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129:34887), c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425)
I20260810 22:56:27.559383 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba153b8da8934c569a002542a0c8c613" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" is_pre_election: true
I20260810 22:56:27.559417 31625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba153b8da8934c569a002542a0c8c613" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41aecfa088e42c3b892498d2be52d8f" is_pre_election: true
I20260810 22:56:27.559556 31625 raft_consensus.cc:2454] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 0.
I20260810 22:56:27.559556 31465 raft_consensus.cc:2454] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 0.
I20260810 22:56:27.559872 31707 leader_election.cc:302] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3000506ef956460eb065859b1b034ac6, c41aecfa088e42c3b892498d2be52d8f; no voters:
I20260810 22:56:27.560088 31880 raft_consensus.cc:2781] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:27.560168 31880 raft_consensus.cc:482] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:27.560211 31880 raft_consensus.cc:3037] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.562202 31880 raft_consensus.cc:504] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.562642 31880 leader_election.cc:288] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 election: Requested vote from peers c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129:34887), c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425)
I20260810 22:56:27.562994 31465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba153b8da8934c569a002542a0c8c613" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c94538c4842140b49cb9ec5e6b4d31a9"
I20260810 22:56:27.563017 31625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba153b8da8934c569a002542a0c8c613" candidate_uuid: "3000506ef956460eb065859b1b034ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41aecfa088e42c3b892498d2be52d8f"
I20260810 22:56:27.563133 31465 raft_consensus.cc:3037] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.563150 31625 raft_consensus.cc:3037] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.564225 31465 raft_consensus.cc:2454] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 1.
I20260810 22:56:27.564587 31706 leader_election.cc:302] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3000506ef956460eb065859b1b034ac6, c94538c4842140b49cb9ec5e6b4d31a9; no voters:
I20260810 22:56:27.564822 31880 raft_consensus.cc:2781] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:27.564833 31625 raft_consensus.cc:2454] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3000506ef956460eb065859b1b034ac6 in term 1.
I20260810 22:56:27.565059 31880 raft_consensus.cc:686] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 1 LEADER]: Becoming Leader. State: Replica: 3000506ef956460eb065859b1b034ac6, State: Running, Role: LEADER
I20260810 22:56:27.565229 31880 consensus_queue.cc:237] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.567071 31283 catalog_manager.cc:5495] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3000506ef956460eb065859b1b034ac6 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "3000506ef956460eb065859b1b034ac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } health_report { overall_health: HEALTHY } } }
Aug 10 22:56:27 dist-test-slave-6nmv krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402584, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:27.627157 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.619433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.29.78.190:40345)
0810 22:56:27.619564 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.619568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.619826 (+ 258us) server_negotiation.cc:407] Connection header received
0810 22:56:27.619927 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.619929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.619966 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.620053 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:27.620729 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.621258 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.621960 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.622145 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.622210 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.624277 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:27.624280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:27.624282 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:27.624302 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:27.625520 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.626030 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.626032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.626033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.626108 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:27.626426 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:27.626428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:27.626429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:27.626581 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:27.626652 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.626945 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.627045 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:56:27.627342 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.619348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40345 (local address 127.0.0.1:39826)
0810 22:56:27.619753 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.619765 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.619857 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.620101 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.620103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.620532 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:27.620624 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.620630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.621340 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.621342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.621868 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.621874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.622229 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.622231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.622286 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.622787 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:27.622797 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:27.624158 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:27.625607 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:27.625645 (+ 38us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:27.625650 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:27.625920 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:27.626196 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:27.626198 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:27.626199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:27.626354 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:27.626666 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:27.626671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:27.626769 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.626892 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.627144 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":68,"thread_start_us":95,"threads_started":1}
I20260810 22:56:27.631175 31283 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:39826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:56:27.631922 31283 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:27.636806 31603 tablet_service.cc:1467] Processing CreateTablet for tablet a14c33c82c934989be582005ab41e3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=213fe012d76040e5a327503836a4ad76]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:27.636905 31443 tablet_service.cc:1467] Processing CreateTablet for tablet a14c33c82c934989be582005ab41e3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=213fe012d76040e5a327503836a4ad76]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:27.637250 31603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c33c82c934989be582005ab41e3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.637257 31443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c33c82c934989be582005ab41e3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.636806 31760 tablet_service.cc:1467] Processing CreateTablet for tablet a14c33c82c934989be582005ab41e3dd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=213fe012d76040e5a327503836a4ad76]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:27.639248 31760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c33c82c934989be582005ab41e3dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:27.640021 31859 tablet_bootstrap.cc:492] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: Bootstrap starting.
I20260810 22:56:27.640120 31860 tablet_bootstrap.cc:492] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: Bootstrap starting.
I20260810 22:56:27.641592 31859 tablet_bootstrap.cc:654] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.641661 31860 tablet_bootstrap.cc:654] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.643088 31858 tablet_bootstrap.cc:492] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: Bootstrap starting.
I20260810 22:56:27.643433 31859 tablet_bootstrap.cc:492] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: No bootstrap required, opened a new log
I20260810 22:56:27.643522 31859 ts_tablet_manager.cc:1397] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 22:56:27.643754 31860 tablet_bootstrap.cc:492] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: No bootstrap required, opened a new log
I20260810 22:56:27.643901 31860 ts_tablet_manager.cc:1397] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260810 22:56:27.644243 31859 raft_consensus.cc:348] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.644351 31858 tablet_bootstrap.cc:654] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:27.644390 31859 raft_consensus.cc:374] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.644426 31859 raft_consensus.cc:729] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41aecfa088e42c3b892498d2be52d8f, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.644570 31859 consensus_queue.cc:260] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.644899 31859 ts_tablet_manager.cc:1428] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 22:56:27.645143 31865 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.645211 31860 raft_consensus.cc:348] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.645328 31860 raft_consensus.cc:374] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.645331 31865 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:27.645366 31860 raft_consensus.cc:729] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94538c4842140b49cb9ec5e6b4d31a9, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.645509 31860 consensus_queue.cc:260] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.645576 31858 tablet_bootstrap.cc:492] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: No bootstrap required, opened a new log
I20260810 22:56:27.645644 31858 ts_tablet_manager.cc:1397] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:27.645802 31864 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.646020 31864 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:27.646086 31858 raft_consensus.cc:348] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.646183 31858 raft_consensus.cc:374] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:27.646214 31858 raft_consensus.cc:729] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3000506ef956460eb065859b1b034ac6, State: Initialized, Role: FOLLOWER
I20260810 22:56:27.646332 31858 consensus_queue.cc:260] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.646553 31860 ts_tablet_manager.cc:1428] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 22:56:27.646576 31858 ts_tablet_manager.cc:1428] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:27.646591 31880 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.646790 31880 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 22:56:27.676050 31676 tablet.cc:2369] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:27.905489 31864 raft_consensus.cc:482] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:27.905642 31864 raft_consensus.cc:504] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.906801 31864 leader_election.cc:288] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425), 3000506ef956460eb065859b1b034ac6 (127.29.78.131:42755)
I20260810 22:56:27.911880 31905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.907169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:50219 (local address 127.29.78.130:33425)
0810 22:56:27.907429 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.907433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.907455 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:56:27.907598 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.907601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.907668 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.907761 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:27.908278 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.908934 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.910586 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.911132 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.911272 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.911324 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.911540 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.911638 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260810 22:56:27.911849 31904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.907082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:33425 (local address 127.29.78.129:50219)
0810 22:56:27.907322 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.907340 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.907445 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.907928 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.907933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.907988 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:27.908123 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.908130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.909038 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.909040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.910449 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.910460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.911270 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.911273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.911386 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.911410 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.911507 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.911565 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"spinlock_wait_cycles":38912,"thread_start_us":83,"threads_started":1}
I20260810 22:56:27.912364 31906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.907441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42755 (local address 127.29.78.129:40055)
0810 22:56:27.907858 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.907871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.907998 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.908341 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.908346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.908370 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:27.908668 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.908683 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.909533 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.909536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.910852 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.910859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.911727 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.911731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.911943 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.911991 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.912083 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.912157 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"spinlock_wait_cycles":5248,"thread_start_us":126,"threads_started":1}
I20260810 22:56:27.912392 31625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c33c82c934989be582005ab41e3dd" candidate_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41aecfa088e42c3b892498d2be52d8f" is_pre_election: true
I20260810 22:56:27.912556 31625 raft_consensus.cc:2454] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94538c4842140b49cb9ec5e6b4d31a9 in term 0.
I20260810 22:56:27.912611 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.907576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:40055 (local address 127.29.78.131:42755)
0810 22:56:27.907940 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.907945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.907960 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:56:27.908125 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.908128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.908205 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.908294 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:27.908801 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.909412 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.910968 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.911603 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.911727 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.911818 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.912259 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.912380 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":64,"threads_started":1}
I20260810 22:56:27.913130 31390 leader_election.cc:302] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c41aecfa088e42c3b892498d2be52d8f, c94538c4842140b49cb9ec5e6b4d31a9; no voters:
I20260810 22:56:27.913285 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c33c82c934989be582005ab41e3dd" candidate_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3000506ef956460eb065859b1b034ac6" is_pre_election: true
I20260810 22:56:27.913451 31864 raft_consensus.cc:2781] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:27.913503 31864 raft_consensus.cc:482] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:27.913530 31864 raft_consensus.cc:3037] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.913528 31780 raft_consensus.cc:2454] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c94538c4842140b49cb9ec5e6b4d31a9 in term 0.
I20260810 22:56:27.914829 31864 raft_consensus.cc:504] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.915132 31864 leader_election.cc:288] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [CANDIDATE]: Term 1 election: Requested vote from peers c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425), 3000506ef956460eb065859b1b034ac6 (127.29.78.131:42755)
I20260810 22:56:27.915539 31625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c33c82c934989be582005ab41e3dd" candidate_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c41aecfa088e42c3b892498d2be52d8f"
I20260810 22:56:27.915550 31780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c33c82c934989be582005ab41e3dd" candidate_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3000506ef956460eb065859b1b034ac6"
I20260810 22:56:27.915666 31625 raft_consensus.cc:3037] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.915678 31780 raft_consensus.cc:3037] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:27.916764 31780 raft_consensus.cc:2454] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94538c4842140b49cb9ec5e6b4d31a9 in term 1.
I20260810 22:56:27.917022 31625 raft_consensus.cc:2454] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94538c4842140b49cb9ec5e6b4d31a9 in term 1.
I20260810 22:56:27.917044 31390 leader_election.cc:302] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3000506ef956460eb065859b1b034ac6, c94538c4842140b49cb9ec5e6b4d31a9; no voters:
I20260810 22:56:27.917249 31864 raft_consensus.cc:2781] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:27.917436 31864 raft_consensus.cc:686] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 LEADER]: Becoming Leader. State: Replica: c94538c4842140b49cb9ec5e6b4d31a9, State: Running, Role: LEADER
I20260810 22:56:27.917695 31864 consensus_queue.cc:237] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } }
I20260810 22:56:27.918972 31908 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: TxnStatusTablet state changed. Reason: New leader c94538c4842140b49cb9ec5e6b4d31a9. Latest consensus state: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.919098 31908 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:27.919427 31910 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:56:27.919618 31910 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:56:27.920167 31284 catalog_manager.cc:5495] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:27.939113 31899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.933355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34887 (local address 127.0.0.1:56628)
0810 22:56:27.933565 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.933579 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.933671 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.934345 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.934349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.934371 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:27.934554 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.934562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.936123 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.936126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.937126 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.937138 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.937760 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.937763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.937914 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.937986 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:27.938776 (+ 790us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:27.938792 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.938855 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.938941 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260810 22:56:27.939669 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.933446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.29.78.129:34887)
0810 22:56:27.934011 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.934025 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.934039 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:27.934103 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.934106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.934164 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.934296 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:27.934768 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.935987 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.937246 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.937624 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.937720 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.938176 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:27.938646 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:27.938825 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.938876 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.938946 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":130,"threads_started":1}
I20260810 22:56:27.943317 31780 raft_consensus.cc:1261] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Refusing update from remote peer c94538c4842140b49cb9ec5e6b4d31a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:27.943912 31908 consensus_queue.cc:1035] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:27.944190 31625 raft_consensus.cc:1261] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Refusing update from remote peer c94538c4842140b49cb9ec5e6b4d31a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:27.944545 31908 consensus_queue.cc:1035] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:27.946061 31865 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: TxnStatusTablet state changed. Reason: New leader c94538c4842140b49cb9ec5e6b4d31a9. Latest consensus state: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.946215 31865 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:27.947345 31880 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: TxnStatusTablet state changed. Reason: New leader c94538c4842140b49cb9ec5e6b4d31a9. Latest consensus state: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.947546 31880 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:27.950078 31864 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.950204 31864 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:27.950333 31864 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.950428 31864 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P c94538c4842140b49cb9ec5e6b4d31a9: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:27.950981 31865 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.951087 31865 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P c41aecfa088e42c3b892498d2be52d8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:27.954036 31880 tablet_replica.cc:439] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 } } peers { permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 } } peers { permanent_uuid: "3000506ef956460eb065859b1b034ac6" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42755 } } }
I20260810 22:56:27.954170 31880 tablet_replica.cc:442] T a14c33c82c934989be582005ab41e3dd P 3000506ef956460eb065859b1b034ac6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:27.965690 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.962339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:43569 (local address 127.29.78.129:34887)
0810 22:56:27.962450 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.962454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.962583 (+ 129us) server_negotiation.cc:407] Connection header received
0810 22:56:27.962739 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.962744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.962848 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.962941 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:27.963170 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.963721 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.964593 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.964819 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.964950 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.965253 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:27.965321 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:27.965402 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.965505 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 22:56:27.965575 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:56:27.965771 31927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.962224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:34887 (local address 127.29.78.131:43569)
0810 22:56:27.962473 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.962489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.962628 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.962949 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.962953 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.962960 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:27.963059 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.963065 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.963831 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.963836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.964469 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.964477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.964962 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.964966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.965064 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.965107 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:27.965412 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:27.965421 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.965562 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 22:56:27.965647 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260810 22:56:27.971961 31625 raft_consensus.cc:1261] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Refusing update from remote peer 3000506ef956460eb065859b1b034ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:27.972023 31465 raft_consensus.cc:1261] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Refusing update from remote peer 3000506ef956460eb065859b1b034ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:27.972361 31880 consensus_queue.cc:1035] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c94538c4842140b49cb9ec5e6b4d31a9" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 34887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:27.972550 31868 consensus_queue.cc:1035] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41aecfa088e42c3b892498d2be52d8f" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 33425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:27.976637 31916 mvcc.cc:205] Tried to move back new op lower bound from 7317105000329486336 to 7317104998668881920. Current Snapshot: MvccSnapshot[applied={T|T < 7317105000329486336}]
I20260810 22:56:28.000164 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.994050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:59975 (local address 127.29.78.131:42755)
0810 22:56:27.994162 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:27.994167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:27.995931 (+ 1764us) server_negotiation.cc:407] Connection header received
0810 22:56:27.996041 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:27.996043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:27.996125 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:27.996188 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:27.996677 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.997547 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.998991 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.999192 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.999267 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.999653 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:27.999724 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:27.999773 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:27.999970 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 22:56:28.000012 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 22:56:28.000576 31932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:27.994258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42755 (local address 127.29.78.129:59975)
0810 22:56:27.995802 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:27.995830 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:27.995951 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:27.996228 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:27.996230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:27.996239 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:27.996572 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.996581 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.997674 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.997677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.998818 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:27.998830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:27.999323 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:27.999327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:27.999476 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:27.999530 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:27.999825 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:27.999848 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:56:27.999937 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 22:56:28.000020 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1327,"thread_start_us":164,"threads_started":1}
I20260810 22:56:28.023871 31284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:28.024039 31284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:28.026260 31284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2ae681dbaed84c0faa359fa0de7c834f: Sending DeleteTablet for 3 replicas of tablet ba153b8da8934c569a002542a0c8c613
I20260810 22:56:28.026866 31760 tablet_service.cc:1514] Processing DeleteTablet for tablet ba153b8da8934c569a002542a0c8c613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:28 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:58086
I20260810 22:56:28.027328 31934 tablet_replica.cc:330] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: stopping tablet replica
I20260810 22:56:28.027573 31934 raft_consensus.cc:2229] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:28.027813 31934 raft_consensus.cc:2258] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:28.028430 31443 tablet_service.cc:1514] Processing DeleteTablet for tablet ba153b8da8934c569a002542a0c8c613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:28 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56612
I20260810 22:56:28.028862 31934 ts_tablet_manager.cc:1905] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:28.028971 31603 tablet_service.cc:1514] Processing DeleteTablet for tablet ba153b8da8934c569a002542a0c8c613 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:28 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36560
I20260810 22:56:28.029706 31936 tablet_replica.cc:330] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: stopping tablet replica
I20260810 22:56:28.029862 31936 raft_consensus.cc:2229] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:28.030010 31936 raft_consensus.cc:2258] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:28.030575 31936 ts_tablet_manager.cc:1905] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:28.032954 31934 ts_tablet_manager.cc:1918] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:28.033077 31934 log.cc:1196] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/ba153b8da8934c569a002542a0c8c613
I20260810 22:56:28.033589 31934 ts_tablet_manager.cc:1939] T ba153b8da8934c569a002542a0c8c613 P 3000506ef956460eb065859b1b034ac6: Deleting consensus metadata
I20260810 22:56:28.034368 31270 catalog_manager.cc:4841] TS 3000506ef956460eb065859b1b034ac6 (127.29.78.131:42755): tablet ba153b8da8934c569a002542a0c8c613 (table test-table [id=4e84f1ffd8024b9ca59c9537cf4c0332]) successfully deleted
I20260810 22:56:28.035187 31936 ts_tablet_manager.cc:1918] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:28.035277 31936 log.cc:1196] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/ba153b8da8934c569a002542a0c8c613
I20260810 22:56:28.035609 31936 ts_tablet_manager.cc:1939] T ba153b8da8934c569a002542a0c8c613 P c94538c4842140b49cb9ec5e6b4d31a9: Deleting consensus metadata
I20260810 22:56:28.036199 31939 tablet_replica.cc:330] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: stopping tablet replica
I20260810 22:56:28.036269 31268 catalog_manager.cc:4841] TS c94538c4842140b49cb9ec5e6b4d31a9 (127.29.78.129:34887): tablet ba153b8da8934c569a002542a0c8c613 (table test-table [id=4e84f1ffd8024b9ca59c9537cf4c0332]) successfully deleted
I20260810 22:56:28.036339 31939 raft_consensus.cc:2229] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:28.036453 31939 raft_consensus.cc:2258] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:28.037422 31939 ts_tablet_manager.cc:1905] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:28.040148 31939 ts_tablet_manager.cc:1918] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:28.040221 31939 log.cc:1196] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/ba153b8da8934c569a002542a0c8c613
I20260810 22:56:28.040494 31939 ts_tablet_manager.cc:1939] T ba153b8da8934c569a002542a0c8c613 P c41aecfa088e42c3b892498d2be52d8f: Deleting consensus metadata
I20260810 22:56:28.041198 31270 catalog_manager.cc:4841] TS c41aecfa088e42c3b892498d2be52d8f (127.29.78.130:33425): tablet ba153b8da8934c569a002542a0c8c613 (table test-table [id=4e84f1ffd8024b9ca59c9537cf4c0332]) successfully deleted
Aug 10 22:56:28 dist-test-slave-6nmv krb5kdc[31232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402587, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:28.050860 31914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:28.029238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56644 (local address 127.29.78.129:34887)
0810 22:56:28.029340 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:28.029344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:28.036834 (+ 7490us) server_negotiation.cc:407] Connection header received
0810 22:56:28.036970 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:28.036973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:28.037021 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:28.037092 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:28.038144 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:28.038858 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:28.039932 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:28.040256 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:28.040369 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:28.047117 (+ 6748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:28.047126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:28.047141 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:28.047167 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:28.049149 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:28.049562 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:28.049568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:28.049571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:28.049659 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:28.049988 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:28.049992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:28.049993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:28.050265 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:28.050458 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:28.050584 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 22:56:28.050717 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 22:56:28.051411 31510 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56644
I20260810 22:56:28.060081 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:28.053041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39838 (local address 127.29.78.190:40345)
0810 22:56:28.053149 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:28.053153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:28.053215 (+ 62us) server_negotiation.cc:407] Connection header received
0810 22:56:28.053314 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:28.053316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:28.053370 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:28.053484 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:28.054187 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:28.054959 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:28.055648 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:28.055946 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:28.056042 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:28.057191 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:28.057195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:28.057197 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:28.057224 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:28.058687 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:28.059038 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:28.059041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:28.059042 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:28.059126 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:28.059346 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:28.059349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:28.059350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:28.059550 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:28.059668 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:28.059838 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 22:56:28.059945 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 22:56:28.060515 31284 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39838
I20260810 22:56:28.061604 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 31367
I20260810 22:56:28.075486 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 31526
I20260810 22:56:28.083375 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 31685
I20260810 22:56:28.091400 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 31248
2026-08-10T22:56:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4459 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:28 dist-test-slave-6nmv krb5kdc[31947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:28 dist-test-slave-6nmv krb5kdc[31947](info): set up 2 sockets
Aug 10 22:56:28 dist-test-slave-6nmv krb5kdc[31947](info): commencing operation
krb5kdc: starting...
W20260810 22:56:29.133288 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:29 dist-test-slave-6nmv krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:29.244760 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37283
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:37283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:29.347323 31961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:29.347573 31961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:29.347637 31961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:29.351400 31961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:29.351470 31961 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:56:29.351498 31961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:29.351568 31961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:29.351653 31961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:29.351711 31961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:29.356570 31961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:37283
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:37283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:29.357483 31961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:29.358381 31961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:29.366255 31968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:29.366230 31967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:29.366370 31961 server_base.cc:1034] running on GCE node
W20260810 22:56:29.366425 31970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:29.367074 31961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:29.367983 31961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:29.369187 31961 hybrid_clock.cc:648] HybridClock initialized: now 1786402589369171 us; error 27 us; skew 500 ppm
Aug 10 22:56:29 dist-test-slave-6nmv krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:29.371948 31961 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:29.372845 31961 webserver.cc:460] Webserver started at http://127.29.78.190:39241/ using document root <none> and password file <none>
I20260810 22:56:29.373345 31961 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:29.373428 31961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:29.373683 31961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:29.375658 31961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "d2b310c8406745ceb78b6e8a986c1529"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "80247892567ea9eb6cf321f39801df7f"
server_key_iv: "99fd38b0e06aa2f09ebe927ca840fbb3"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.376181 31961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "d2b310c8406745ceb78b6e8a986c1529"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "80247892567ea9eb6cf321f39801df7f"
server_key_iv: "99fd38b0e06aa2f09ebe927ca840fbb3"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.379709 31961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:29.381943 31978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:29.383057 31961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:29.383195 31961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "d2b310c8406745ceb78b6e8a986c1529"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "80247892567ea9eb6cf321f39801df7f"
server_key_iv: "99fd38b0e06aa2f09ebe927ca840fbb3"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.383298 31961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:29.414778 31961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:29.417740 31961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:29.417964 31961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:29.425886 31961 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:37283
I20260810 22:56:29.425912 32049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:37283 every 8 connection(s)
I20260810 22:56:29.426801 31961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:29.429433 32050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:29.430545 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 31961
I20260810 22:56:29.430630 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:29.430907 30010 external_mini_cluster.cc:1428] Setting key aa0e52b87c5483c146d90bd9b22bf555
I20260810 22:56:29.435521 32050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Bootstrap starting.
I20260810 22:56:29.437853 32050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:29.438577 32050 log.cc:824] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:56:29 dist-test-slave-6nmv krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:29.440658 32050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: No bootstrap required, opened a new log
I20260810 22:56:29.443434 32050 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b310c8406745ceb78b6e8a986c1529" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37283 } }
I20260810 22:56:29.443699 32050 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:29.443809 32050 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b310c8406745ceb78b6e8a986c1529, State: Initialized, Role: FOLLOWER
I20260810 22:56:29.444288 32050 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b310c8406745ceb78b6e8a986c1529" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37283 } }
I20260810 22:56:29.444478 32050 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:29.444581 32050 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:29.444713 32050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:29.444970 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.432365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.29.78.190:37283)
0810 22:56:29.432775 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:29.432789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:29.432834 (+ 45us) server_negotiation.cc:407] Connection header received
0810 22:56:29.433404 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:29.433419 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:29.433744 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:29.434131 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:29.434942 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.435880 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.436642 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.437074 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.437237 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.439839 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:29.439843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:29.439854 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:29.439878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:29.442104 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.442504 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.442509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.442513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.442609 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.442887 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.442891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.442892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.443242 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:29.443453 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:29.443740 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 22:56:29.444037 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260810 22:56:29.445931 32050 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b310c8406745ceb78b6e8a986c1529" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37283 } }
I20260810 22:56:29.446286 32050 leader_election.cc:302] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2b310c8406745ceb78b6e8a986c1529; no voters:
I20260810 22:56:29.446635 32050 leader_election.cc:288] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:29.446708 32057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:29.446976 32057 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [term 1 LEADER]: Becoming Leader. State: Replica: d2b310c8406745ceb78b6e8a986c1529, State: Running, Role: LEADER
I20260810 22:56:29.447613 32057 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b310c8406745ceb78b6e8a986c1529" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37283 } }
I20260810 22:56:29.447886 32050 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:29.448740 32057 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2b310c8406745ceb78b6e8a986c1529. Latest consensus state: current_term: 1 leader_uuid: "d2b310c8406745ceb78b6e8a986c1529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b310c8406745ceb78b6e8a986c1529" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37283 } } }
I20260810 22:56:29.448892 32057 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:29.449141 32058 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2b310c8406745ceb78b6e8a986c1529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2b310c8406745ceb78b6e8a986c1529" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 37283 } } }
I20260810 22:56:29.449239 32058 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529 [sys.catalog]: This master's current role is: LEADER
W20260810 22:56:29.452867 32072 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:56:29.452935 32072 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:56:29.453042 32066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:29.453778 32066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:29.459321 32066 catalog_manager.cc:1347] Generated new cluster ID: f419ba6a6f054bc3965421b5d079014a
I20260810 22:56:29.459398 32066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:29.472632 32066 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:29.473572 32066 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:29.481096 32066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Generated new TSK 0
I20260810 22:56:29.481725 32066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:29.511056 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37283
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:29.620843 32078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:29.621073 32078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:29.621119 32078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:29.624833 32078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:29.624883 32078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:29.624919 32078 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:56:29.624933 32078 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:29.624962 32078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:29.629479 32078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:29.630311 32078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:29.631160 32078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:29.640306 32084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:29.640367 32083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:29.640524 32078 server_base.cc:1034] running on GCE node
W20260810 22:56:29.640708 32088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:29.641060 32078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:29.641628 32078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:29.642808 32078 hybrid_clock.cc:648] HybridClock initialized: now 1786402589642809 us; error 57 us; skew 500 ppm
Aug 10 22:56:29 dist-test-slave-6nmv krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:29.645421 32078 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:29.646212 32078 webserver.cc:460] Webserver started at http://127.29.78.129:41697/ using document root <none> and password file <none>
I20260810 22:56:29.646715 32078 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:29.646798 32078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:29.647008 32078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:29.648988 32078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "ddbc409f08094fbf94bef416b37bea28"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "d57152c145e2988b208fddabb9371726"
server_key_iv: "56e7617e4f90a50023b2b4f2621fc5d5"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.649524 32078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "ddbc409f08094fbf94bef416b37bea28"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "d57152c145e2988b208fddabb9371726"
server_key_iv: "56e7617e4f90a50023b2b4f2621fc5d5"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.652838 32078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:56:29.655179 32095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:29.656224 32078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:29.656344 32078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "ddbc409f08094fbf94bef416b37bea28"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "d57152c145e2988b208fddabb9371726"
server_key_iv: "56e7617e4f90a50023b2b4f2621fc5d5"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.656442 32078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:29.684131 32078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:29.687623 32078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:29.687834 32078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:29.689177 32078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:29.689252 32078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:29.689317 32078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:29.689364 32078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:29 dist-test-slave-6nmv krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:29.701040 32078 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:40943
I20260810 22:56:29.701112 32227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:40943 every 8 connection(s)
I20260810 22:56:29.702106 32078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:29.703831 32104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.690551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37283 (local address 127.29.78.129:33531)
0810 22:56:29.691015 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:29.691056 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:29.691948 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:29.692809 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:29.692819 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:29.693316 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:29.693997 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.694018 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.694916 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.694920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.695841 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.695851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.696527 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.696532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.696733 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.697372 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:29.697396 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:29.699230 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:29.701205 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.701210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.701222 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.701511 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.701781 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.701784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.701786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.701955 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.702390 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:29.702397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:29.702685 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 22:56:29.702875 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 22:56:29.703158 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":132,"threads_started":1}
I20260810 22:56:29.704473 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.690060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33531 (local address 127.29.78.190:37283)
0810 22:56:29.690185 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:29.690188 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:29.691153 (+ 965us) server_negotiation.cc:407] Connection header received
0810 22:56:29.692198 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:29.692204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:29.692269 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:29.692333 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:29.694170 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.694773 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.695983 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.696342 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.696468 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.699387 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:29.699393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:29.699400 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:29.699435 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:29.701079 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.701606 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.701609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.701610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.701685 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.702069 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.702071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.702072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.702292 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:29.702368 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:29.704233 (+ 1865us) server_negotiation.cc:299] Negotiation successful
0810 22:56:29.704364 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:56:29.705456 32228 heartbeater.cc:347] Connected to a master server at 127.29.78.190:37283
I20260810 22:56:29.705600 32228 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:29.706102 32228 heartbeater.cc:511] Master 127.29.78.190:37283 requested a full tablet report, sending...
I20260810 22:56:29.707472 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 32078
I20260810 22:56:29.707577 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:29.707585 31997 ts_manager.cc:194] Registered new tserver with Master: ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943)
I20260810 22:56:29.707850 30010 external_mini_cluster.cc:1428] Setting key ff5b78eb6fc8b2a10aa5f781931d3d0c
I20260810 22:56:29.708890 31997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:33531
I20260810 22:56:29.715706 32235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.707536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37283 (local address 127.29.78.129:58073)
0810 22:56:29.707806 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:29.707820 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:29.707908 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:29.708125 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:29.708127 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:29.708336 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:29.708455 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.708462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.709640 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.709644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.710540 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.710555 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.711091 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.711097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.711216 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.711731 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:29.711750 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:29.712207 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:29.714178 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.714181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.714182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.714377 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.714686 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.714688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.714689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.714775 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.715229 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:29.715231 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:29.715351 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 22:56:29.715460 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 22:56:29.715563 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260810 22:56:29.715847 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.707574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:58073 (local address 127.29.78.190:37283)
0810 22:56:29.707670 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:29.707673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:29.707900 (+ 227us) server_negotiation.cc:407] Connection header received
0810 22:56:29.707976 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:29.707978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:29.708021 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:29.708097 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:29.708611 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.709508 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.710687 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.710969 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.711071 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.712360 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:29.712364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:29.712369 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:29.712398 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:29.714062 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.714482 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.714488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.714493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.714593 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.714863 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.714869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.714873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.715131 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:29.715240 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:29.715561 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 22:56:29.715680 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:29.736713 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37283
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:29.844821 32239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:29.845093 32239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:29.845198 32239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:29.849056 32239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:29.849152 32239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:29.849221 32239 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:56:29.849273 32239 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:29.849362 32239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:29.854504 32239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:29.855568 32239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:29.856599 32239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:29.866043 32244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:29.866043 32245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:29.866084 32247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:29.866086 32239 server_base.cc:1034] running on GCE node
I20260810 22:56:29.866643 32239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:29.867141 32239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:29.868347 32239 hybrid_clock.cc:648] HybridClock initialized: now 1786402589868330 us; error 31 us; skew 500 ppm
Aug 10 22:56:29 dist-test-slave-6nmv krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:29.870999 32239 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:29.871850 32239 webserver.cc:460] Webserver started at http://127.29.78.130:42623/ using document root <none> and password file <none>
I20260810 22:56:29.872396 32239 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:29.872488 32239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:29.872710 32239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:29.874812 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "a5c1f99a9fea69fa4f4fd20383e2bcec"
server_key_iv: "4f830ccf953b81f20d6ebbbec3c3defe"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.875403 32239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "a5c1f99a9fea69fa4f4fd20383e2bcec"
server_key_iv: "4f830ccf953b81f20d6ebbbec3c3defe"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.879071 32239 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:56:29.881285 32256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:29.882529 32239 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:29.882671 32239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
format_stamp: "Formatted at 2026-08-10 22:56:29 on dist-test-slave-6nmv"
server_key: "a5c1f99a9fea69fa4f4fd20383e2bcec"
server_key_iv: "4f830ccf953b81f20d6ebbbec3c3defe"
server_key_version: "encryptionkey@0"
I20260810 22:56:29.882777 32239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:29.901686 32239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:29.904950 32239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:29.905143 32239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:29.906647 32239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:29.906725 32239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:29.906782 32239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:29.906813 32239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:29 dist-test-slave-6nmv krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:29.917662 32239 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:40405
I20260810 22:56:29.917725 32388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:40405 every 8 connection(s)
I20260810 22:56:29.918751 32239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:29.921116 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.907503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:42143 (local address 127.29.78.190:37283)
0810 22:56:29.907616 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:29.907619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:29.908649 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 22:56:29.909761 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:29.909767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:29.909888 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:29.910001 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:29.911345 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.912251 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.914157 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.914516 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.914611 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.917580 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:29.917584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:29.917586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:29.917617 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:29.919004 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.919622 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.919625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.919627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.919714 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.920061 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.920065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.920066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.920253 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:29.920360 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:29.920822 (+ 462us) server_negotiation.cc:299] Negotiation successful
0810 22:56:29.920951 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 22:56:29.921887 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.907998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37283 (local address 127.29.78.130:42143)
0810 22:56:29.908494 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:29.908535 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:29.909467 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:29.910177 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:29.910184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:29.910605 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:29.911196 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.911225 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.912400 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.912404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.914009 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.914024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.914684 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.914689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.914923 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.915520 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:29.915544 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:29.917418 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:29.919137 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.919142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.919155 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.919505 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.919807 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.919810 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.919812 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.919953 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.920362 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:29.920368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:29.920601 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 22:56:29.920779 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 22:56:29.921079 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":164,"threads_started":1}
I20260810 22:56:29.922854 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 32239
I20260810 22:56:29.922969 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:29.923089 32389 heartbeater.cc:347] Connected to a master server at 127.29.78.190:37283
I20260810 22:56:29.923219 32389 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:29.923296 30010 external_mini_cluster.cc:1428] Setting key 8febd3b0b5c043d06565f829a9c896c6
I20260810 22:56:29.923789 32389 heartbeater.cc:511] Master 127.29.78.190:37283 requested a full tablet report, sending...
I20260810 22:56:29.925602 31997 ts_manager.cc:194] Registered new tserver with Master: 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405)
I20260810 22:56:29.926414 31997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:42143
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:56:29.933911 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.925461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37283 (local address 127.29.78.130:56765)
0810 22:56:29.925737 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:29.925750 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:29.925866 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:29.927511 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:29.927514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:29.927713 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:29.927809 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.927814 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.928532 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.928535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.929072 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:29.929079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.929554 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.929557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:29.929653 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.930253 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:29.930265 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:29.930586 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:29.932335 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.932340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.932342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.932548 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.932835 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:29.932850 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:29.932852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:29.932936 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:29.933336 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:29.933339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:29.933424 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 22:56:29.933536 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 22:56:29.933688 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260810 22:56:29.933902 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:29.927189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:56765 (local address 127.29.78.190:37283)
0810 22:56:29.927320 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:29.927324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:29.927342 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:56:29.927370 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:29.927371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:29.927403 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:29.927495 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:29.927923 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.928411 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.929187 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:29.929430 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:29.929548 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:29.930728 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:29.930731 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:29.930734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:29.930756 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:29.932223 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.932651 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.932656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.932657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.932742 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:29.933026 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:29.933030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:29.933032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:29.933243 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:29.933346 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:29.933610 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 22:56:29.933722 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:29.952267 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:37283
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:30.060761 32401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:30.061015 32401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:30.061096 32401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:30.065060 32401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:30.065115 32401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:30.065177 32401 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 22:56:30.065212 32401 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:30.065276 32401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:30.070457 32401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:30.071553 32401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:30.072613 32401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:30.082131 32407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:30.082131 32410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:30.082155 32408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:30.082396 32401 server_base.cc:1034] running on GCE node
I20260810 22:56:30.082832 32401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:30.083434 32401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:30.084596 32401 hybrid_clock.cc:648] HybridClock initialized: now 1786402590084578 us; error 28 us; skew 500 ppm
Aug 10 22:56:30 dist-test-slave-6nmv krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402590, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:30.087476 32401 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:30.088299 32401 webserver.cc:460] Webserver started at http://127.29.78.131:37735/ using document root <none> and password file <none>
I20260810 22:56:30.088800 32401 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:30.088857 32401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:30.089025 32401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:30.091037 32401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "7e740c72a9e04824820bbc12ceffa197"
format_stamp: "Formatted at 2026-08-10 22:56:30 on dist-test-slave-6nmv"
server_key: "6d298cfff12dfc96654fa28a43ce4d2d"
server_key_iv: "b42c76a61944c5ba1f992d386738b5d4"
server_key_version: "encryptionkey@0"
I20260810 22:56:30.091538 32401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "7e740c72a9e04824820bbc12ceffa197"
format_stamp: "Formatted at 2026-08-10 22:56:30 on dist-test-slave-6nmv"
server_key: "6d298cfff12dfc96654fa28a43ce4d2d"
server_key_iv: "b42c76a61944c5ba1f992d386738b5d4"
server_key_version: "encryptionkey@0"
I20260810 22:56:30.095036 32401 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:56:30.097174 32419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:30.098381 32401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:30.098474 32401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "7e740c72a9e04824820bbc12ceffa197"
format_stamp: "Formatted at 2026-08-10 22:56:30 on dist-test-slave-6nmv"
server_key: "6d298cfff12dfc96654fa28a43ce4d2d"
server_key_iv: "b42c76a61944c5ba1f992d386738b5d4"
server_key_version: "encryptionkey@0"
I20260810 22:56:30.098556 32401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:30.116513 32401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:30.119230 32401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:30.119370 32401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:30.120613 32401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:30.120659 32401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:30.120719 32401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:30.120754 32401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:30 dist-test-slave-6nmv krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402590, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:30.131350 32401 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:46555
I20260810 22:56:30.131424 32549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:46555 every 8 connection(s)
I20260810 22:56:30.132460 32401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:30.135509 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.121540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:43251 (local address 127.29.78.190:37283)
0810 22:56:30.121668 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.121672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.122652 (+ 980us) server_negotiation.cc:407] Connection header received
0810 22:56:30.123803 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.123807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.123885 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.123949 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:30.125621 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.126558 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.128345 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.128703 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.128844 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.131650 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:30.131655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:30.131658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:30.131685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:30.133385 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.133930 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.133933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.133934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.134214 (+ 280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.134585 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.134588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.134589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.134776 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:30.134854 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.135281 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.135393 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:56:30.136279 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.122049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37283 (local address 127.29.78.131:43251)
0810 22:56:30.122501 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.122540 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.123540 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.124352 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.124362 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.124797 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:30.125444 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.125466 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.126722 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.126728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.128199 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.128210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.128867 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.128870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.129008 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.129644 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:30.129669 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:30.131465 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:30.133497 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:30.133501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:30.133511 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:30.133760 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:30.134321 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:30.134324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:30.134326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:30.134487 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:30.134868 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:30.134873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:30.135074 (+ 201us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.135216 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.135462 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260810 22:56:30.137458 32550 heartbeater.cc:347] Connected to a master server at 127.29.78.190:37283
I20260810 22:56:30.137612 32550 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:30.138111 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 32401
I20260810 22:56:30.138181 32550 heartbeater.cc:511] Master 127.29.78.190:37283 requested a full tablet report, sending...
I20260810 22:56:30.138232 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:30.138556 30010 external_mini_cluster.cc:1428] Setting key 4703a6d5db07d6bc4f6588a069e46707
I20260810 22:56:30.139281 31999 ts_manager.cc:194] Registered new tserver with Master: 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555)
I20260810 22:56:30.140050 31999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:43251
I20260810 22:56:30.140591 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:56:30.149966 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.140585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:36799 (local address 127.29.78.190:37283)
0810 22:56:30.142179 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.142183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.142194 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:56:30.142221 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.142223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.142255 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.142317 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:30.142906 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.143739 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.144854 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.145055 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.145127 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.146428 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:30.146431 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:30.146433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:30.146453 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:30.148287 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.148791 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.148794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.148795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.148864 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.149250 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.149253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.149254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.149413 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:30.149491 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.149711 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.149795 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1512}
I20260810 22:56:30.149988 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.140534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37283 (local address 127.29.78.131:36799)
0810 22:56:30.140771 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.140783 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.140851 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.142370 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.142383 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.142619 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:30.142767 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.142777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.143861 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.143863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.144712 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.144727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.145164 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.145170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.145298 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.145855 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:30.145874 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:30.146299 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:30.148402 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:30.148407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:30.148411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:30.148696 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:30.148972 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:30.148975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:30.148977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:30.149145 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:30.149516 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:30.149518 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:30.149593 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.149677 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.149775 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":92,"threads_started":1}
Aug 10 22:56:30 dist-test-slave-6nmv krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:56:30 dist-test-slave-6nmv krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:30.168875 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.161185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44722 (local address 127.29.78.190:37283)
0810 22:56:30.161303 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.161306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.161317 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:56:30.161350 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.161352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.161386 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.161447 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:30.162247 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.162736 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.163555 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.163792 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.163864 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.166236 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:30.166239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:30.166241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:30.166263 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:30.167464 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.167829 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.167831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.167832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.167919 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.168137 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.168139 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.168140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.168293 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:30.168391 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.168580 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.168667 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:56:30.171850 31997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:30.174924 31997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:30.189046 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.183346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.29.78.130:40405)
0810 22:56:30.184748 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.184759 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.184778 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:56:30.184836 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.184854 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.184995 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.185159 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:30.185510 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.186449 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.187885 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.188403 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.188519 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.188680 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.188824 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.188916 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1278,"thread_start_us":128,"threads_started":1}
I20260810 22:56:30.189036 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.183148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:40405 (local address 127.0.0.1:56888)
0810 22:56:30.184063 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.184117 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.184246 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.185241 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.185243 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.185257 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:30.185364 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.185370 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.186590 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.186593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.187754 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.187763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.188520 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.188523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.188620 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.188655 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.188793 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.188901 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"spinlock_wait_cycles":1408,"thread_start_us":81,"threads_started":1}
I20260810 22:56:30.189718 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.183345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40943 (local address 127.0.0.1:46608)
0810 22:56:30.184348 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.184383 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.184462 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.185166 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.185172 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.185204 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:30.185357 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.185363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.187047 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.187050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.188391 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.188402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.189364 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.189368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.189445 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.189472 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.189522 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.189581 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":99,"spinlock_wait_cycles":6272,"thread_start_us":100,"threads_started":1}
I20260810 22:56:30.190325 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.184041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.29.78.131:46555)
0810 22:56:30.184548 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.184558 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.184646 (+ 88us) server_negotiation.cc:407] Connection header received
0810 22:56:30.184754 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.184762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.184977 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.185160 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:30.185721 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.186935 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.188536 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.189235 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.189341 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.189478 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.190104 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.190196 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":114,"threads_started":1}
I20260810 22:56:30.190351 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.184000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.29.78.129:40943)
0810 22:56:30.184749 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.184763 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.184783 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:56:30.184862 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.184866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.185009 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.185518 (+ 509us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:30.185700 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.186835 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.188527 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.189236 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.189927 (+ 691us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.190110 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.190153 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.190226 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":92,"threads_started":1}
I20260810 22:56:30.190567 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.183554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46555 (local address 127.0.0.1:57368)
0810 22:56:30.184536 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.184553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.184645 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.185164 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.185170 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.185205 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:30.185606 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.185612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.187049 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.187051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.188392 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.188402 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.189542 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.189545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.189623 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.189642 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.189702 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.189763 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":444,"spinlock_wait_cycles":3712,"thread_start_us":133,"threads_started":1}
I20260810 22:56:30.192173 32314 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9e21fda4c54ae6b335ad232e0fd422 (DEFAULT_TABLE table=test-table [id=56b5725c4c01455bb3d3ebae78559b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:30.192528 32151 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9e21fda4c54ae6b335ad232e0fd422 (DEFAULT_TABLE table=test-table [id=56b5725c4c01455bb3d3ebae78559b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:30.193346 32314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9e21fda4c54ae6b335ad232e0fd422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.193095 32473 tablet_service.cc:1467] Processing CreateTablet for tablet 5b9e21fda4c54ae6b335ad232e0fd422 (DEFAULT_TABLE table=test-table [id=56b5725c4c01455bb3d3ebae78559b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:30.193471 32151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9e21fda4c54ae6b335ad232e0fd422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.194029 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b9e21fda4c54ae6b335ad232e0fd422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.199714 32574 tablet_bootstrap.cc:492] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Bootstrap starting.
I20260810 22:56:30.201248 32576 tablet_bootstrap.cc:492] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Bootstrap starting.
I20260810 22:56:30.201249 32575 tablet_bootstrap.cc:492] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Bootstrap starting.
I20260810 22:56:30.202142 32574 tablet_bootstrap.cc:654] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.203243 32575 tablet_bootstrap.cc:654] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.203409 32574 log.cc:824] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:30.203446 32576 tablet_bootstrap.cc:654] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.204134 32575 log.cc:824] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:30.204167 32576 log.cc:824] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:30.205351 32574 tablet_bootstrap.cc:492] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: No bootstrap required, opened a new log
I20260810 22:56:30.205560 32574 ts_tablet_manager.cc:1397] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:56:30.205993 32576 tablet_bootstrap.cc:492] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: No bootstrap required, opened a new log
I20260810 22:56:30.206180 32576 ts_tablet_manager.cc:1397] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:56:30.207177 32575 tablet_bootstrap.cc:492] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: No bootstrap required, opened a new log
I20260810 22:56:30.207374 32575 ts_tablet_manager.cc:1397] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:56:30.209451 32576 raft_consensus.cc:348] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.209484 32574 raft_consensus.cc:348] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.209702 32576 raft_consensus.cc:374] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.209717 32574 raft_consensus.cc:374] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.209764 32574 raft_consensus.cc:729] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.209784 32576 raft_consensus.cc:729] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e740c72a9e04824820bbc12ceffa197, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.210237 32576 consensus_queue.cc:260] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.210273 32574 consensus_queue.cc:260] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.210891 32550 heartbeater.cc:503] Master 127.29.78.190:37283 was elected leader, sending a full tablet report...
I20260810 22:56:30.211205 32576 ts_tablet_manager.cc:1428] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:56:30.211143 32575 raft_consensus.cc:348] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.211409 32575 raft_consensus.cc:374] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.211480 32575 raft_consensus.cc:729] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049c0e83e7f24a2ea4e098dbc700bc56, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.211766 32228 heartbeater.cc:503] Master 127.29.78.190:37283 was elected leader, sending a full tablet report...
I20260810 22:56:30.211917 32575 consensus_queue.cc:260] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.211854 32574 ts_tablet_manager.cc:1428] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 22:56:30.212617 32389 heartbeater.cc:503] Master 127.29.78.190:37283 was elected leader, sending a full tablet report...
I20260810 22:56:30.212778 32575 ts_tablet_manager.cc:1428] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:56:30.282717 32582 raft_consensus.cc:482] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:30.282969 32582 raft_consensus.cc:504] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.284178 32582 leader_election.cc:288] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405), 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555)
I20260810 22:56:30.288739 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.284579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:51801 (local address 127.29.78.131:46555)
0810 22:56:30.284706 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.284711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.284897 (+ 186us) server_negotiation.cc:407] Connection header received
0810 22:56:30.285052 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.285056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.285122 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.285199 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:30.285538 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.286123 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.287786 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.288169 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.288259 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.288335 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.288539 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.288625 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:56:30.288740 32585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.284447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46555 (local address 127.29.78.129:51801)
0810 22:56:30.284812 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.284823 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.284938 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.285212 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.285214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.285239 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:30.285395 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.285400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.286252 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.286277 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.287676 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.287686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.288297 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.288300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.288383 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.288414 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.288481 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.288583 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":54,"spinlock_wait_cycles":92032,"thread_start_us":95,"threads_started":1}
I20260810 22:56:30.288771 32584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.284417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:40405 (local address 127.29.78.129:32841)
0810 22:56:30.284737 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.284763 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.284896 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.285184 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.285187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.285202 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:30.285378 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.285389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.286320 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.286323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.287676 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.287686 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.288352 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.288358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.288491 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.288519 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.288590 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.288649 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":114,"thread_start_us":90,"threads_started":1}
I20260810 22:56:30.288800 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.284566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:32841 (local address 127.29.78.130:40405)
0810 22:56:30.284706 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.284711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.284849 (+ 138us) server_negotiation.cc:407] Connection header received
0810 22:56:30.285013 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.285022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.285064 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.285161 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:30.285521 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.286187 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.287834 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.288201 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.288335 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.288386 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.288619 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.288690 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:56:30.289527 32338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9e21fda4c54ae6b335ad232e0fd422" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" is_pre_election: true
I20260810 22:56:30.289546 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9e21fda4c54ae6b335ad232e0fd422" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e740c72a9e04824820bbc12ceffa197" is_pre_election: true
I20260810 22:56:30.289834 32498 raft_consensus.cc:2454] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 0.
I20260810 22:56:30.289855 32338 raft_consensus.cc:2454] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 0.
I20260810 22:56:30.290337 32099 leader_election.cc:302] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e740c72a9e04824820bbc12ceffa197, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:30.290604 32582 raft_consensus.cc:2781] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:30.290712 32582 raft_consensus.cc:482] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:30.290771 32582 raft_consensus.cc:3037] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.291810 32582 raft_consensus.cc:504] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.292193 32582 leader_election.cc:288] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 election: Requested vote from peers 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405), 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555)
I20260810 22:56:30.292529 32338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9e21fda4c54ae6b335ad232e0fd422" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
I20260810 22:56:30.292583 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b9e21fda4c54ae6b335ad232e0fd422" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e740c72a9e04824820bbc12ceffa197"
I20260810 22:56:30.292678 32498 raft_consensus.cc:3037] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.292663 32338 raft_consensus.cc:3037] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.294117 32498 raft_consensus.cc:2454] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 1.
I20260810 22:56:30.294426 32099 leader_election.cc:302] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e740c72a9e04824820bbc12ceffa197, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:30.294567 32338 raft_consensus.cc:2454] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 1.
I20260810 22:56:30.294765 32582 raft_consensus.cc:2781] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:30.295051 32582 raft_consensus.cc:686] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 1 LEADER]: Becoming Leader. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Running, Role: LEADER
I20260810 22:56:30.295364 32582 consensus_queue.cc:237] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.298507 31997 catalog_manager.cc:5495] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddbc409f08094fbf94bef416b37bea28 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "ddbc409f08094fbf94bef416b37bea28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } health_report { overall_health: UNKNOWN } } }
Aug 10 22:56:30 dist-test-slave-6nmv krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402589, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:30.331313 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.323598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:37283 (local address 127.0.0.1:44734)
0810 22:56:30.323936 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.323949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.324084 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.324374 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.324376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.324791 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:30.324902 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.324909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.325630 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.325632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.326213 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.326222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.326622 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.326625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.326686 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.327243 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:30.327255 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:30.328623 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:30.330066 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:30.330074 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:30.330078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:30.330287 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:30.330500 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:30.330502 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:30.330503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:30.330576 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:30.330883 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:30.330888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:30.330980 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.331061 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.331172 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":110,"threads_started":1}
I20260810 22:56:30.331326 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.323651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44734 (local address 127.29.78.190:37283)
0810 22:56:30.323803 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.323808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.324030 (+ 222us) server_negotiation.cc:407] Connection header received
0810 22:56:30.324219 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.324221 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.324263 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.324330 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:30.325018 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.325495 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.326326 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.326511 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.326579 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.328746 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:30.328749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:30.328750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:30.328770 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:30.329979 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.330364 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.330367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.330368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.330431 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:30.330643 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:30.330645 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:30.330646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:30.330792 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:30.330857 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.331118 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.331221 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":45}
I20260810 22:56:30.335156 31997 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:44734:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:56:30.335865 31997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:30.341046 32473 tablet_service.cc:1467] Processing CreateTablet for tablet a7f11524b6ca46b4904f20fa3077ff1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=138cdf29a8054475ae3272f1588a85f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:30.340850 32151 tablet_service.cc:1467] Processing CreateTablet for tablet a7f11524b6ca46b4904f20fa3077ff1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=138cdf29a8054475ae3272f1588a85f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:30.341646 32151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f11524b6ca46b4904f20fa3077ff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.341646 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f11524b6ca46b4904f20fa3077ff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.344712 32576 tablet_bootstrap.cc:492] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: Bootstrap starting.
I20260810 22:56:30.344712 32574 tablet_bootstrap.cc:492] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: Bootstrap starting.
I20260810 22:56:30.345136 32314 tablet_service.cc:1467] Processing CreateTablet for tablet a7f11524b6ca46b4904f20fa3077ff1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=138cdf29a8054475ae3272f1588a85f7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:30.345681 32314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7f11524b6ca46b4904f20fa3077ff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.345685 32576 tablet_bootstrap.cc:654] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.345685 32574 tablet_bootstrap.cc:654] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.347106 32576 tablet_bootstrap.cc:492] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: No bootstrap required, opened a new log
I20260810 22:56:30.347204 32576 ts_tablet_manager.cc:1397] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 22:56:30.347357 32574 tablet_bootstrap.cc:492] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: No bootstrap required, opened a new log
I20260810 22:56:30.347427 32574 ts_tablet_manager.cc:1397] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 22:56:30.347730 32576 raft_consensus.cc:348] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.347829 32576 raft_consensus.cc:374] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.347865 32576 raft_consensus.cc:729] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e740c72a9e04824820bbc12ceffa197, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.347848 32574 raft_consensus.cc:348] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.347957 32574 raft_consensus.cc:374] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.347985 32574 raft_consensus.cc:729] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.347986 32576 consensus_queue.cc:260] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.348094 32574 consensus_queue.cc:260] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.348363 32582 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.348507 32582 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:30.348850 32574 ts_tablet_manager.cc:1428] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:56:30.349563 32580 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.349852 32580 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:30.350163 32575 tablet_bootstrap.cc:492] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: Bootstrap starting.
I20260810 22:56:30.351349 32575 tablet_bootstrap.cc:654] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.353504 32575 tablet_bootstrap.cc:492] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: No bootstrap required, opened a new log
I20260810 22:56:30.353592 32575 ts_tablet_manager.cc:1397] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:30.354136 32575 raft_consensus.cc:348] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.354252 32575 raft_consensus.cc:374] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.354288 32575 raft_consensus.cc:729] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049c0e83e7f24a2ea4e098dbc700bc56, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.354447 32575 consensus_queue.cc:260] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.354748 32575 ts_tablet_manager.cc:1428] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:30.355712 32576 ts_tablet_manager.cc:1428] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20260810 22:56:30.356284 32583 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.356401 32583 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:30.364151 32580 raft_consensus.cc:482] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:30.364235 32580 raft_consensus.cc:504] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.365244 32580 leader_election.cc:288] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943), 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405)
I20260810 22:56:30.368958 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.365437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40943 (local address 127.29.78.131:51411)
0810 22:56:30.365585 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.365600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.365682 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.366065 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.366068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.366085 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:30.366201 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.366206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.366921 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.366927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.367695 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.367705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.368542 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.368545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.368644 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.368670 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.368751 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.368817 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":29}
I20260810 22:56:30.368959 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.365680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:51411 (local address 127.29.78.129:40943)
0810 22:56:30.365876 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.365887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.365898 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:56:30.365933 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.365935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.365969 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.366054 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:30.366304 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.366809 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.367849 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.368428 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.368551 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.368602 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.368765 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.368822 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 22:56:30.368976 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.365680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:35931 (local address 127.29.78.130:40405)
0810 22:56:30.365800 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.365806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.365975 (+ 169us) server_negotiation.cc:407] Connection header received
0810 22:56:30.366113 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.366116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.366159 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.366235 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:30.366484 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.366939 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.367969 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.368470 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.368594 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.368638 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.368811 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.368869 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:56:30.368997 32600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.365600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:40405 (local address 127.29.78.131:35931)
0810 22:56:30.365893 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.365903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.366012 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.366245 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.366247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.366262 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:30.366384 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.366389 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.367057 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.367061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.367872 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.367882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.368599 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.368603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.368691 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.368710 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.368795 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.368861 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":115,"threads_started":1}
I20260810 22:56:30.369524 32338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f11524b6ca46b4904f20fa3077ff1d" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" is_pre_election: true
I20260810 22:56:30.369591 32172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f11524b6ca46b4904f20fa3077ff1d" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbc409f08094fbf94bef416b37bea28" is_pre_election: true
I20260810 22:56:30.369655 32338 raft_consensus.cc:2454] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 0.
I20260810 22:56:30.369835 32172 raft_consensus.cc:2454] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 0.
I20260810 22:56:30.369984 32420 leader_election.cc:302] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049c0e83e7f24a2ea4e098dbc700bc56, 7e740c72a9e04824820bbc12ceffa197; no voters:
I20260810 22:56:30.370258 32580 raft_consensus.cc:2781] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:30.370328 32580 raft_consensus.cc:482] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:30.370375 32580 raft_consensus.cc:3037] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.371260 32580 raft_consensus.cc:504] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.371609 32580 leader_election.cc:288] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 election: Requested vote from peers ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943), 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405)
I20260810 22:56:30.371953 32172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f11524b6ca46b4904f20fa3077ff1d" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbc409f08094fbf94bef416b37bea28"
I20260810 22:56:30.371984 32338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7f11524b6ca46b4904f20fa3077ff1d" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
I20260810 22:56:30.372076 32338 raft_consensus.cc:3037] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.372076 32172 raft_consensus.cc:3037] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.373152 32338 raft_consensus.cc:2454] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 1.
I20260810 22:56:30.373152 32172 raft_consensus.cc:2454] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 1.
I20260810 22:56:30.373426 32420 leader_election.cc:302] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049c0e83e7f24a2ea4e098dbc700bc56, 7e740c72a9e04824820bbc12ceffa197; no voters:
I20260810 22:56:30.373598 32580 raft_consensus.cc:2781] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:30.373899 32580 raft_consensus.cc:686] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [term 1 LEADER]: Becoming Leader. State: Replica: 7e740c72a9e04824820bbc12ceffa197, State: Running, Role: LEADER
I20260810 22:56:30.374250 32580 consensus_queue.cc:237] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.376725 32601 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: TxnStatusTablet state changed. Reason: New leader 7e740c72a9e04824820bbc12ceffa197. Latest consensus state: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.376832 32601 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:30.377116 32604 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:56:30.377327 32604 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:56:30.377965 31993 catalog_manager.cc:5495] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } health_report { overall_health: HEALTHY } } }
W20260810 22:56:30.383723 32551 tablet.cc:2369] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:30.390130 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.386738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57374 (local address 127.29.78.131:46555)
0810 22:56:30.386879 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.386884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.386897 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:56:30.387058 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.387061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.387120 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.387245 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:30.387493 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.388008 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.388783 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.389000 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.389100 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.389376 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:30.389729 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:30.389870 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.389939 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.389985 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 22:56:30.390151 32594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.386646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46555 (local address 127.0.0.1:57374)
0810 22:56:30.386789 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.386805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.386922 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.387287 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.387289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.387297 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:30.387395 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.387401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.388111 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.388114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.388668 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.388676 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.389102 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.389105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.389199 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.389237 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:30.389841 (+ 604us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:30.389857 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.389932 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.390014 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 22:56:30.395282 32338 raft_consensus.cc:1261] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Refusing update from remote peer 7e740c72a9e04824820bbc12ceffa197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:30.395381 32172 raft_consensus.cc:1261] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Refusing update from remote peer 7e740c72a9e04824820bbc12ceffa197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:30.395916 32580 consensus_queue.cc:1035] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:30.396304 32580 consensus_queue.cc:1035] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:30.399410 32582 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: TxnStatusTablet state changed. Reason: New leader 7e740c72a9e04824820bbc12ceffa197. Latest consensus state: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.399519 32582 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:30.400317 32583 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: TxnStatusTablet state changed. Reason: New leader 7e740c72a9e04824820bbc12ceffa197. Latest consensus state: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.400434 32583 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:30.403549 32582 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.403687 32582 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P ddbc409f08094fbf94bef416b37bea28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:30.403825 32583 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.403918 32583 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P 049c0e83e7f24a2ea4e098dbc700bc56: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:30.406257 32616 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.406379 32616 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:30.406545 32616 tablet_replica.cc:439] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } } }
I20260810 22:56:30.406623 32616 tablet_replica.cc:442] T a7f11524b6ca46b4904f20fa3077ff1d P 7e740c72a9e04824820bbc12ceffa197: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:30.409332 32609 mvcc.cc:205] Tried to move back new op lower bound from 7317105010252107776 to 7317105010175758336. Current Snapshot: MvccSnapshot[applied={T|T < 7317105010252107776 or (T in {7317105010252107776})}]
I20260810 22:56:30.418103 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.414482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.29.78.129:40943)
0810 22:56:30.414591 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.414595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.414608 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:56:30.414754 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.414756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.414820 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.414911 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:30.415170 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.415674 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.416806 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.417028 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.417103 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.417379 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:30.417747 (+ 368us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:30.417875 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.417939 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.417980 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 22:56:30.419865 32390 tablet.cc:2369] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:30.429733 32573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.425609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52531 (local address 127.29.78.131:46555)
0810 22:56:30.425723 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.425727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.425980 (+ 253us) server_negotiation.cc:407] Connection header received
0810 22:56:30.426126 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.426129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.426181 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.426275 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:30.426574 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.427325 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.428514 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.428851 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.428949 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.429275 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:30.429357 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:30.429432 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.429546 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.429597 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:30.429774 32621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.425495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46555 (local address 127.29.78.129:52531)
0810 22:56:30.425889 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.425903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.426014 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.426312 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.426315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.426323 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:30.426451 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.426457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.427450 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.427454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.428393 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.428403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.428965 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.428968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.429069 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.429118 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:30.429451 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:30.429458 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.429523 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.429601 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":107,"threads_started":1}
I20260810 22:56:30.437490 32338 raft_consensus.cc:1261] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Refusing update from remote peer ddbc409f08094fbf94bef416b37bea28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:30.437636 32498 raft_consensus.cc:1261] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Refusing update from remote peer ddbc409f08094fbf94bef416b37bea28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:30.437999 32586 consensus_queue.cc:1035] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:30.438194 32582 consensus_queue.cc:1035] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:30.442483 32610 mvcc.cc:205] Tried to move back new op lower bound from 7317105010427826176 to 7317105009852747776. Current Snapshot: MvccSnapshot[applied={T|T < 7317105010427826176}]
I20260810 22:56:30.442934 32608 mvcc.cc:205] Tried to move back new op lower bound from 7317105010427826176 to 7317105009852747776. Current Snapshot: MvccSnapshot[applied={T|T < 7317105010427826176}]
I20260810 22:56:30.459029 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.455628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:41409 (local address 127.29.78.129:40943)
0810 22:56:30.455752 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:30.455757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:30.455960 (+ 203us) server_negotiation.cc:407] Connection header received
0810 22:56:30.456016 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:30.456017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:30.456054 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:30.456111 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:30.456374 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.456813 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.457927 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.458231 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.458314 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.458620 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:30.458683 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:30.458733 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:30.458844 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 22:56:30.458932 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:56:30.459041 32557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:30.455571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40943 (local address 127.29.78.131:41409)
0810 22:56:30.455806 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:30.455819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:30.455903 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:30.456151 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:30.456154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:30.456161 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:30.456277 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.456283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.456915 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.456918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.457761 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:30.457771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:30.458384 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:30.458387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:30.458468 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:30.458515 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:30.458771 (+ 256us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:30.458776 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:56:30.458839 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 22:56:30.458915 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 22:56:30.482174 31997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:30.482371 31997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:30.485147 31997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Sending DeleteTablet for 3 replicas of tablet 5b9e21fda4c54ae6b335ad232e0fd422
I20260810 22:56:30.485896 32151 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9e21fda4c54ae6b335ad232e0fd422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:30 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:46608
I20260810 22:56:30.486106 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9e21fda4c54ae6b335ad232e0fd422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:30 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260810 22:56:30.486569 32633 tablet_replica.cc:330] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: stopping tablet replica
I20260810 22:56:30.486924 32633 raft_consensus.cc:2229] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:30.486960 31997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:56:30.487200 32633 raft_consensus.cc:2258] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 22:56:30.487349 31997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:30.488727 32633 ts_tablet_manager.cc:1905] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:30.489568 32314 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b9e21fda4c54ae6b335ad232e0fd422 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:30 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56888
I20260810 22:56:30.492077 32633 ts_tablet_manager.cc:1918] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:30.492172 32633 log.cc:1196] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/5b9e21fda4c54ae6b335ad232e0fd422
I20260810 22:56:30.492499 32633 ts_tablet_manager.cc:1939] T 5b9e21fda4c54ae6b335ad232e0fd422 P 7e740c72a9e04824820bbc12ceffa197: Deleting consensus metadata
I20260810 22:56:30.493541 32473 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b437f3a5242ca997322d701177187 (DEFAULT_TABLE table=test-table [id=0cce2e426f7f48e5b606b5bdb9bbc246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:30.493718 32314 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b437f3a5242ca997322d701177187 (DEFAULT_TABLE table=test-table [id=0cce2e426f7f48e5b606b5bdb9bbc246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:30.493810 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b437f3a5242ca997322d701177187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.493997 32634 tablet_replica.cc:330] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: stopping tablet replica
I20260810 22:56:30.494033 32314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b437f3a5242ca997322d701177187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.494097 32151 tablet_service.cc:1467] Processing CreateTablet for tablet 4a8b437f3a5242ca997322d701177187 (DEFAULT_TABLE table=test-table [id=0cce2e426f7f48e5b606b5bdb9bbc246]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:30.494311 32634 raft_consensus.cc:2229] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:30.494352 32151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a8b437f3a5242ca997322d701177187. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:30.494556 32634 raft_consensus.cc:2258] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:30.496006 32576 tablet_bootstrap.cc:492] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: Bootstrap starting.
I20260810 22:56:30.496721 32575 tablet_bootstrap.cc:492] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: Bootstrap starting.
I20260810 22:56:30.496975 32576 tablet_bootstrap.cc:654] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.497695 32575 tablet_bootstrap.cc:654] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.498113 32631 tablet_replica.cc:330] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: stopping tablet replica
I20260810 22:56:30.498345 32574 tablet_bootstrap.cc:492] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: Bootstrap starting.
I20260810 22:56:30.498518 32631 raft_consensus.cc:2229] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:30.498898 32631 raft_consensus.cc:2258] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:30.499248 32574 tablet_bootstrap.cc:654] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:30.500782 31982 catalog_manager.cc:4841] TS 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555): tablet 5b9e21fda4c54ae6b335ad232e0fd422 (table test-table [id=56b5725c4c01455bb3d3ebae78559b7a]) successfully deleted
I20260810 22:56:30.501152 32631 ts_tablet_manager.cc:1905] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:30.501909 32574 tablet_bootstrap.cc:492] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: No bootstrap required, opened a new log
I20260810 22:56:30.501992 32574 ts_tablet_manager.cc:1397] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260810 22:56:30.502592 32574 raft_consensus.cc:348] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.502734 32574 raft_consensus.cc:374] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.502774 32574 raft_consensus.cc:729] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.503010 32634 ts_tablet_manager.cc:1905] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:30.503072 32574 consensus_queue.cc:260] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.503305 32576 tablet_bootstrap.cc:492] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: No bootstrap required, opened a new log
I20260810 22:56:30.503368 32574 ts_tablet_manager.cc:1428] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:30.503401 32576 ts_tablet_manager.cc:1397] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260810 22:56:30.503840 32576 raft_consensus.cc:348] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.503964 32576 raft_consensus.cc:374] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.504010 32576 raft_consensus.cc:729] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e740c72a9e04824820bbc12ceffa197, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.504143 32576 consensus_queue.cc:260] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.504431 32576 ts_tablet_manager.cc:1428] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:30.505641 32575 tablet_bootstrap.cc:492] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: No bootstrap required, opened a new log
I20260810 22:56:30.505728 32575 ts_tablet_manager.cc:1397] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 22:56:30.506227 32575 raft_consensus.cc:348] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.506340 32575 raft_consensus.cc:374] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:30.506383 32575 raft_consensus.cc:729] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049c0e83e7f24a2ea4e098dbc700bc56, State: Initialized, Role: FOLLOWER
I20260810 22:56:30.506309 32631 ts_tablet_manager.cc:1918] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:30.506538 32631 log.cc:1196] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/5b9e21fda4c54ae6b335ad232e0fd422
I20260810 22:56:30.506522 32575 consensus_queue.cc:260] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.506811 32575 ts_tablet_manager.cc:1428] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:30.506858 32631 ts_tablet_manager.cc:1939] T 5b9e21fda4c54ae6b335ad232e0fd422 P ddbc409f08094fbf94bef416b37bea28: Deleting consensus metadata
I20260810 22:56:30.505722 32634 ts_tablet_manager.cc:1918] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:30.507391 32634 log.cc:1196] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/5b9e21fda4c54ae6b335ad232e0fd422
I20260810 22:56:30.507736 32634 ts_tablet_manager.cc:1939] T 5b9e21fda4c54ae6b335ad232e0fd422 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting consensus metadata
I20260810 22:56:30.508669 31981 catalog_manager.cc:4841] TS ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943): tablet 5b9e21fda4c54ae6b335ad232e0fd422 (table test-table [id=56b5725c4c01455bb3d3ebae78559b7a]) successfully deleted
I20260810 22:56:30.508950 31979 catalog_manager.cc:4841] TS 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405): tablet 5b9e21fda4c54ae6b335ad232e0fd422 (table test-table [id=56b5725c4c01455bb3d3ebae78559b7a]) successfully deleted
I20260810 22:56:30.574208 32582 raft_consensus.cc:482] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:30.574393 32582 raft_consensus.cc:504] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.574892 32582 leader_election.cc:288] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405), 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555)
I20260810 22:56:30.575294 32338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b437f3a5242ca997322d701177187" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" is_pre_election: true
I20260810 22:56:30.575378 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b437f3a5242ca997322d701177187" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e740c72a9e04824820bbc12ceffa197" is_pre_election: true
I20260810 22:56:30.575445 32338 raft_consensus.cc:2454] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 0.
I20260810 22:56:30.575556 32498 raft_consensus.cc:2454] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 0.
I20260810 22:56:30.575750 32096 leader_election.cc:302] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049c0e83e7f24a2ea4e098dbc700bc56, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:30.575908 32582 raft_consensus.cc:2781] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:30.575969 32582 raft_consensus.cc:482] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:30.576015 32582 raft_consensus.cc:3037] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.576977 32582 raft_consensus.cc:504] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.577298 32582 leader_election.cc:288] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 election: Requested vote from peers 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405), 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555)
I20260810 22:56:30.577618 32338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b437f3a5242ca997322d701177187" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
I20260810 22:56:30.577618 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a8b437f3a5242ca997322d701177187" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e740c72a9e04824820bbc12ceffa197"
I20260810 22:56:30.577726 32498 raft_consensus.cc:3037] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.577726 32338 raft_consensus.cc:3037] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:30.578816 32498 raft_consensus.cc:2454] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 1.
I20260810 22:56:30.578816 32338 raft_consensus.cc:2454] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 1.
I20260810 22:56:30.579097 32096 leader_election.cc:302] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049c0e83e7f24a2ea4e098dbc700bc56, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:30.579270 32582 raft_consensus.cc:2781] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:30.579387 32582 raft_consensus.cc:686] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 1 LEADER]: Becoming Leader. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Running, Role: LEADER
I20260810 22:56:30.579535 32582 consensus_queue.cc:237] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:30.581137 31997 catalog_manager.cc:5495] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddbc409f08094fbf94bef416b37bea28 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "ddbc409f08094fbf94bef416b37bea28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:30.614562 32498 raft_consensus.cc:1261] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Refusing update from remote peer ddbc409f08094fbf94bef416b37bea28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:30.614683 32338 raft_consensus.cc:1261] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Refusing update from remote peer ddbc409f08094fbf94bef416b37bea28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:30.615123 32627 consensus_queue.cc:1035] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:30.615309 32582 consensus_queue.cc:1035] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:56:30.630086 32610 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 22:56:32.181782 31997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:32.182168 31997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:32.184551 31997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Sending DeleteTablet for 3 replicas of tablet 4a8b437f3a5242ca997322d701177187
I20260810 22:56:32.185113 32151 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a8b437f3a5242ca997322d701177187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:32 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:46608
I20260810 22:56:32.185132 32314 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a8b437f3a5242ca997322d701177187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:32 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56888
I20260810 22:56:32.185288 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a8b437f3a5242ca997322d701177187 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:32 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260810 22:56:32.185503 32669 tablet_replica.cc:330] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: stopping tablet replica
I20260810 22:56:32.185669 32668 tablet_replica.cc:330] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: stopping tablet replica
I20260810 22:56:32.185796 32668 raft_consensus.cc:2229] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:32.185914 32668 raft_consensus.cc:2258] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:32.185911 32669 raft_consensus.cc:2229] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:32.186116 32669 raft_consensus.cc:2258] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:32.188305 32669 ts_tablet_manager.cc:1905] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:32.188319 32668 ts_tablet_manager.cc:1905] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:32.188555 32670 tablet_replica.cc:330] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: stopping tablet replica
I20260810 22:56:32.188690 32670 raft_consensus.cc:2229] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:32.188799 32670 raft_consensus.cc:2258] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:32.189321 32670 ts_tablet_manager.cc:1905] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:32.190902 32668 ts_tablet_manager.cc:1918] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:56:32.191056 32668 log.cc:1196] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/4a8b437f3a5242ca997322d701177187
I20260810 22:56:32.191176 32669 ts_tablet_manager.cc:1918] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:56:32.191251 32669 log.cc:1196] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/4a8b437f3a5242ca997322d701177187
I20260810 22:56:32.191509 32668 ts_tablet_manager.cc:1939] T 4a8b437f3a5242ca997322d701177187 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting consensus metadata
I20260810 22:56:32.191536 32669 ts_tablet_manager.cc:1939] T 4a8b437f3a5242ca997322d701177187 P ddbc409f08094fbf94bef416b37bea28: Deleting consensus metadata
I20260810 22:56:32.192019 32670 ts_tablet_manager.cc:1918] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:56:32.192102 32670 log.cc:1196] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/4a8b437f3a5242ca997322d701177187
I20260810 22:56:32.192274 31981 catalog_manager.cc:4841] TS ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943): tablet 4a8b437f3a5242ca997322d701177187 (table test-table [id=0cce2e426f7f48e5b606b5bdb9bbc246]) successfully deleted
I20260810 22:56:32.192378 32670 ts_tablet_manager.cc:1939] T 4a8b437f3a5242ca997322d701177187 P 7e740c72a9e04824820bbc12ceffa197: Deleting consensus metadata
I20260810 22:56:32.192407 31979 catalog_manager.cc:4841] TS 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405): tablet 4a8b437f3a5242ca997322d701177187 (table test-table [id=0cce2e426f7f48e5b606b5bdb9bbc246]) successfully deleted
I20260810 22:56:32.193049 31982 catalog_manager.cc:4841] TS 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555): tablet 4a8b437f3a5242ca997322d701177187 (table test-table [id=0cce2e426f7f48e5b606b5bdb9bbc246]) successfully deleted
Aug 10 22:56:32 dist-test-slave-6nmv krb5kdc[31947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:56:32 dist-test-slave-6nmv krb5kdc[31947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:32.215261 32678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:32.206792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.29.78.190:37283)
0810 22:56:32.207107 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:32.207115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:32.207134 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:56:32.207280 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:32.207283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:32.207337 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:32.207442 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:32.208148 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.208705 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.209516 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.209736 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.209887 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:32.212314 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:32.212317 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:32.212318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:32.212341 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:32.213663 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:32.214124 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:32.214126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:32.214127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:32.214205 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:32.214445 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:32.214447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:32.214448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:32.214617 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:32.214718 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:32.214921 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 22:56:32.215050 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":107,"threads_started":1}
I20260810 22:56:32.218019 31997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:32.218489 31997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:32.223414 32151 tablet_service.cc:1467] Processing CreateTablet for tablet f033e730cd64494e8aa659f2c852dc24 (DEFAULT_TABLE table=test-table [id=53b16e3bd71c4f63a2269d83d4929b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:32.223492 32473 tablet_service.cc:1467] Processing CreateTablet for tablet f033e730cd64494e8aa659f2c852dc24 (DEFAULT_TABLE table=test-table [id=53b16e3bd71c4f63a2269d83d4929b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:32.223770 32151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f033e730cd64494e8aa659f2c852dc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:32.223852 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f033e730cd64494e8aa659f2c852dc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:32.225920 32679 tablet_bootstrap.cc:492] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Bootstrap starting.
I20260810 22:56:32.225934 32680 tablet_bootstrap.cc:492] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: Bootstrap starting.
I20260810 22:56:32.226150 32314 tablet_service.cc:1467] Processing CreateTablet for tablet f033e730cd64494e8aa659f2c852dc24 (DEFAULT_TABLE table=test-table [id=53b16e3bd71c4f63a2269d83d4929b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:32.226426 32314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f033e730cd64494e8aa659f2c852dc24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:32.226883 32679 tablet_bootstrap.cc:654] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:32.227070 32680 tablet_bootstrap.cc:654] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:32.228083 32679 tablet_bootstrap.cc:492] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: No bootstrap required, opened a new log
I20260810 22:56:32.228188 32679 ts_tablet_manager.cc:1397] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 22:56:32.228637 32679 raft_consensus.cc:348] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.228518 32680 tablet_bootstrap.cc:492] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: No bootstrap required, opened a new log
I20260810 22:56:32.228753 32679 raft_consensus.cc:374] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:32.228786 32679 raft_consensus.cc:729] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Initialized, Role: FOLLOWER
I20260810 22:56:32.228809 32680 ts_tablet_manager.cc:1397] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:32.228904 32679 consensus_queue.cc:260] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.229199 32680 raft_consensus.cc:348] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.229307 32680 raft_consensus.cc:374] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:32.229338 32680 raft_consensus.cc:729] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e740c72a9e04824820bbc12ceffa197, State: Initialized, Role: FOLLOWER
I20260810 22:56:32.229440 32683 tablet_bootstrap.cc:492] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: Bootstrap starting.
I20260810 22:56:32.229461 32680 consensus_queue.cc:260] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.229722 32680 ts_tablet_manager.cc:1428] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:56:32.229722 32679 ts_tablet_manager.cc:1428] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 22:56:32.230463 32683 tablet_bootstrap.cc:654] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:32.232748 32683 tablet_bootstrap.cc:492] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: No bootstrap required, opened a new log
I20260810 22:56:32.232877 32683 ts_tablet_manager.cc:1397] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:32.233232 32683 raft_consensus.cc:348] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.233337 32683 raft_consensus.cc:374] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:32.233378 32683 raft_consensus.cc:729] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049c0e83e7f24a2ea4e098dbc700bc56, State: Initialized, Role: FOLLOWER
I20260810 22:56:32.233507 32683 consensus_queue.cc:260] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.233920 32683 ts_tablet_manager.cc:1428] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 22:56:32.427937 32658 raft_consensus.cc:482] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:32.428110 32658 raft_consensus.cc:504] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.428566 32658 leader_election.cc:288] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405), 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555)
I20260810 22:56:32.428996 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f033e730cd64494e8aa659f2c852dc24" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e740c72a9e04824820bbc12ceffa197" is_pre_election: true
I20260810 22:56:32.429019 32336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f033e730cd64494e8aa659f2c852dc24" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" is_pre_election: true
I20260810 22:56:32.429147 32498 raft_consensus.cc:2454] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 0.
I20260810 22:56:32.429170 32336 raft_consensus.cc:2454] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 0.
I20260810 22:56:32.429471 32099 leader_election.cc:302] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e740c72a9e04824820bbc12ceffa197, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:32.429697 32658 raft_consensus.cc:2781] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:32.429740 32658 raft_consensus.cc:482] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:32.429765 32658 raft_consensus.cc:3037] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:32.430791 32658 raft_consensus.cc:504] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.431167 32658 leader_election.cc:288] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 election: Requested vote from peers 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405), 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555)
I20260810 22:56:32.431478 32498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f033e730cd64494e8aa659f2c852dc24" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e740c72a9e04824820bbc12ceffa197"
I20260810 22:56:32.431506 32336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f033e730cd64494e8aa659f2c852dc24" candidate_uuid: "ddbc409f08094fbf94bef416b37bea28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
I20260810 22:56:32.431623 32498 raft_consensus.cc:3037] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:32.431640 32336 raft_consensus.cc:3037] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:32.432528 32498 raft_consensus.cc:2454] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 1.
I20260810 22:56:32.432813 32336 raft_consensus.cc:2454] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddbc409f08094fbf94bef416b37bea28 in term 1.
I20260810 22:56:32.432837 32099 leader_election.cc:302] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e740c72a9e04824820bbc12ceffa197, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:32.433012 32658 raft_consensus.cc:2781] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:32.433089 32658 raft_consensus.cc:686] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 1 LEADER]: Becoming Leader. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Running, Role: LEADER
I20260810 22:56:32.433224 32658 consensus_queue.cc:237] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.434863 31993 catalog_manager.cc:5495] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddbc409f08094fbf94bef416b37bea28 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "ddbc409f08094fbf94bef416b37bea28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } health_report { overall_health: UNKNOWN } } }
W20260810 22:56:32.457409 32229 tablet.cc:2369] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:32.470894 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:32.467483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.29.78.129:40943)
0810 22:56:32.467781 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:32.467790 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:32.467804 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:32.467865 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:32.467867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:32.467910 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:32.467985 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:32.468241 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.468755 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.469961 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.470183 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.470258 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:32.470466 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:32.470530 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:32.470579 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:32.470676 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 22:56:32.470720 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":121,"threads_started":1}
I20260810 22:56:32.476986 32498 raft_consensus.cc:1261] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Refusing update from remote peer ddbc409f08094fbf94bef416b37bea28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:32.477070 32336 raft_consensus.cc:1261] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Refusing update from remote peer ddbc409f08094fbf94bef416b37bea28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:32.477347 32662 consensus_queue.cc:1035] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:32.477468 32658 consensus_queue.cc:1035] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:32.506992 31993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:32.507220 31993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:32.509402 31993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Sending DeleteTablet for 3 replicas of tablet f033e730cd64494e8aa659f2c852dc24
I20260810 22:56:32.509922 32151 tablet_service.cc:1514] Processing DeleteTablet for tablet f033e730cd64494e8aa659f2c852dc24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:32 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:46608
I20260810 22:56:32.510027 32314 tablet_service.cc:1514] Processing DeleteTablet for tablet f033e730cd64494e8aa659f2c852dc24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:32 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56888
I20260810 22:56:32.510214 32669 tablet_replica.cc:330] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: stopping tablet replica
I20260810 22:56:32.510216 32668 tablet_replica.cc:330] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: stopping tablet replica
I20260810 22:56:32.510254 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet f033e730cd64494e8aa659f2c852dc24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:32 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260810 22:56:32.510334 32668 raft_consensus.cc:2229] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:32.510339 32669 raft_consensus.cc:2229] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:32.510419 32670 tablet_replica.cc:330] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: stopping tablet replica
I20260810 22:56:32.510452 32668 raft_consensus.cc:2258] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:32.510517 32669 raft_consensus.cc:2258] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:32.510533 32670 raft_consensus.cc:2229] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:32.510628 32670 raft_consensus.cc:2258] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:32.511160 32668 ts_tablet_manager.cc:1905] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:32.511271 32670 ts_tablet_manager.cc:1905] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:32.511373 32669 ts_tablet_manager.cc:1905] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:32.511526 31993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:32.512063 31993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:32.513844 32669 ts_tablet_manager.cc:1918] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:32.513844 32670 ts_tablet_manager.cc:1918] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:32.513927 32669 log.cc:1196] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/f033e730cd64494e8aa659f2c852dc24
I20260810 22:56:32.513927 32670 log.cc:1196] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/f033e730cd64494e8aa659f2c852dc24
I20260810 22:56:32.513962 32668 ts_tablet_manager.cc:1918] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:32.514037 32668 log.cc:1196] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/f033e730cd64494e8aa659f2c852dc24
I20260810 22:56:32.514209 32670 ts_tablet_manager.cc:1939] T f033e730cd64494e8aa659f2c852dc24 P 7e740c72a9e04824820bbc12ceffa197: Deleting consensus metadata
I20260810 22:56:32.514209 32669 ts_tablet_manager.cc:1939] T f033e730cd64494e8aa659f2c852dc24 P ddbc409f08094fbf94bef416b37bea28: Deleting consensus metadata
I20260810 22:56:32.514350 32668 ts_tablet_manager.cc:1939] T f033e730cd64494e8aa659f2c852dc24 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting consensus metadata
I20260810 22:56:32.514827 31982 catalog_manager.cc:4841] TS 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555): tablet f033e730cd64494e8aa659f2c852dc24 (table test-table [id=53b16e3bd71c4f63a2269d83d4929b7a]) successfully deleted
I20260810 22:56:32.514883 31981 catalog_manager.cc:4841] TS ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943): tablet f033e730cd64494e8aa659f2c852dc24 (table test-table [id=53b16e3bd71c4f63a2269d83d4929b7a]) successfully deleted
I20260810 22:56:32.515372 31979 catalog_manager.cc:4841] TS 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405): tablet f033e730cd64494e8aa659f2c852dc24 (table test-table [id=53b16e3bd71c4f63a2269d83d4929b7a]) successfully deleted
I20260810 22:56:32.517709 32151 tablet_service.cc:1467] Processing CreateTablet for tablet 2791cceab9894d63ae1780ee054d2c62 (DEFAULT_TABLE table=test-table [id=e764817d7ba644048b7f24a3cae082b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:32.517761 32314 tablet_service.cc:1467] Processing CreateTablet for tablet 2791cceab9894d63ae1780ee054d2c62 (DEFAULT_TABLE table=test-table [id=e764817d7ba644048b7f24a3cae082b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:32.517900 32473 tablet_service.cc:1467] Processing CreateTablet for tablet 2791cceab9894d63ae1780ee054d2c62 (DEFAULT_TABLE table=test-table [id=e764817d7ba644048b7f24a3cae082b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:32.517977 32151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2791cceab9894d63ae1780ee054d2c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:32.518038 32314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2791cceab9894d63ae1780ee054d2c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:32.518128 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2791cceab9894d63ae1780ee054d2c62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:32.519647 32683 tablet_bootstrap.cc:492] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: Bootstrap starting.
I20260810 22:56:32.520046 32679 tablet_bootstrap.cc:492] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28: Bootstrap starting.
I20260810 22:56:32.520529 32683 tablet_bootstrap.cc:654] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:32.520743 32679 tablet_bootstrap.cc:654] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:32.521317 32680 tablet_bootstrap.cc:492] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: Bootstrap starting.
I20260810 22:56:32.521763 32679 tablet_bootstrap.cc:492] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28: No bootstrap required, opened a new log
I20260810 22:56:32.521883 32679 ts_tablet_manager.cc:1397] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 22:56:32.522014 32680 tablet_bootstrap.cc:654] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:32.522185 32679 raft_consensus.cc:348] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.522262 32679 raft_consensus.cc:374] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:32.522281 32679 raft_consensus.cc:729] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbc409f08094fbf94bef416b37bea28, State: Initialized, Role: FOLLOWER
I20260810 22:56:32.522408 32679 consensus_queue.cc:260] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.522734 32679 ts_tablet_manager.cc:1428] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:32.523172 32680 tablet_bootstrap.cc:492] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: No bootstrap required, opened a new log
I20260810 22:56:32.523268 32680 ts_tablet_manager.cc:1397] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:32.523651 32680 raft_consensus.cc:348] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.523741 32680 raft_consensus.cc:374] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:32.523761 32680 raft_consensus.cc:729] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e740c72a9e04824820bbc12ceffa197, State: Initialized, Role: FOLLOWER
I20260810 22:56:32.523901 32680 consensus_queue.cc:260] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.523994 32683 tablet_bootstrap.cc:492] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: No bootstrap required, opened a new log
I20260810 22:56:32.524085 32683 ts_tablet_manager.cc:1397] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 22:56:32.524175 32680 ts_tablet_manager.cc:1428] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:32.524533 32683 raft_consensus.cc:348] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.524652 32683 raft_consensus.cc:374] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:32.524700 32683 raft_consensus.cc:729] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049c0e83e7f24a2ea4e098dbc700bc56, State: Initialized, Role: FOLLOWER
I20260810 22:56:32.524830 32683 consensus_queue.cc:260] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.525131 32683 ts_tablet_manager.cc:1428] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:56:32.647645 32653 raft_consensus.cc:482] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:32.647768 32653 raft_consensus.cc:504] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.648180 32653 leader_election.cc:288] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943), 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405)
I20260810 22:56:32.648556 32172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2791cceab9894d63ae1780ee054d2c62" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbc409f08094fbf94bef416b37bea28" is_pre_election: true
I20260810 22:56:32.648656 32336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2791cceab9894d63ae1780ee054d2c62" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" is_pre_election: true
I20260810 22:56:32.648711 32172 raft_consensus.cc:2454] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 0.
I20260810 22:56:32.648787 32336 raft_consensus.cc:2454] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 0.
I20260810 22:56:32.649020 32422 leader_election.cc:302] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e740c72a9e04824820bbc12ceffa197, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:32.649176 32653 raft_consensus.cc:2781] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:32.649233 32653 raft_consensus.cc:482] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:32.649266 32653 raft_consensus.cc:3037] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:32.650751 32653 raft_consensus.cc:504] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.651070 32653 leader_election.cc:288] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 election: Requested vote from peers ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943), 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405)
I20260810 22:56:32.651434 32172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2791cceab9894d63ae1780ee054d2c62" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddbc409f08094fbf94bef416b37bea28"
I20260810 22:56:32.651470 32336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2791cceab9894d63ae1780ee054d2c62" candidate_uuid: "7e740c72a9e04824820bbc12ceffa197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049c0e83e7f24a2ea4e098dbc700bc56"
I20260810 22:56:32.651546 32172 raft_consensus.cc:3037] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:32.651679 32336 raft_consensus.cc:3037] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:32.652559 32172 raft_consensus.cc:2454] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 1.
I20260810 22:56:32.652652 32336 raft_consensus.cc:2454] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e740c72a9e04824820bbc12ceffa197 in term 1.
I20260810 22:56:32.652917 32422 leader_election.cc:302] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e740c72a9e04824820bbc12ceffa197, ddbc409f08094fbf94bef416b37bea28; no voters:
I20260810 22:56:32.653126 32653 raft_consensus.cc:2781] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:32.653215 32653 raft_consensus.cc:686] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 1 LEADER]: Becoming Leader. State: Replica: 7e740c72a9e04824820bbc12ceffa197, State: Running, Role: LEADER
I20260810 22:56:32.653375 32653 consensus_queue.cc:237] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } }
I20260810 22:56:32.655076 31999 catalog_manager.cc:5495] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "7e740c72a9e04824820bbc12ceffa197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e740c72a9e04824820bbc12ceffa197" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46555 } health_report { overall_health: HEALTHY } } }
I20260810 22:56:32.680007 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:32.677342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57386 (local address 127.29.78.131:46555)
0810 22:56:32.677623 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:32.677629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:32.677639 (+ 10us) server_negotiation.cc:407] Connection header received
0810 22:56:32.677672 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:32.677674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:32.677708 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:32.677772 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:32.677990 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.678474 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.679180 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.679363 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.679429 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:32.679616 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:32.679675 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:32.679722 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:32.679792 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 22:56:32.679829 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":136,"threads_started":1}
I20260810 22:56:32.687979 32697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:32.685002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:48637 (local address 127.29.78.131:46555)
0810 22:56:32.685119 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:32.685123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:32.685251 (+ 128us) server_negotiation.cc:407] Connection header received
0810 22:56:32.685325 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:32.685327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:32.685371 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:32.685422 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:32.685617 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.686191 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.687077 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.687317 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.687390 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:32.687616 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:32.687670 (+ 54us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:32.687761 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:32.687835 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 22:56:32.687871 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":58}
I20260810 22:56:32.688014 32701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:32.684904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46555 (local address 127.29.78.131:48637)
0810 22:56:32.685180 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:32.685191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:32.685267 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:32.685443 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:32.685445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:32.685450 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:32.685541 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:32.685544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.686320 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.686323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:32.686899 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:32.686907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:32.687428 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:32.687431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:32.687497 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:32.687523 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:32.687764 (+ 241us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:32.687769 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:56:32.687816 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 22:56:32.687885 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":12,"thread_start_us":144,"threads_started":1}
I20260810 22:56:32.693343 32336 raft_consensus.cc:1261] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Refusing update from remote peer 7e740c72a9e04824820bbc12ceffa197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:32.693395 32172 raft_consensus.cc:1261] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28 [term 1 FOLLOWER]: Refusing update from remote peer 7e740c72a9e04824820bbc12ceffa197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:32.693780 32653 consensus_queue.cc:1035] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049c0e83e7f24a2ea4e098dbc700bc56" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:32.694015 32616 consensus_queue.cc:1035] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddbc409f08094fbf94bef416b37bea28" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:56:32.706460 32645 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 22:56:34.258422 31993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:34.258630 31993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:34.261221 31993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2b310c8406745ceb78b6e8a986c1529: Sending DeleteTablet for 3 replicas of tablet 2791cceab9894d63ae1780ee054d2c62
I20260810 22:56:34.261828 32151 tablet_service.cc:1514] Processing DeleteTablet for tablet 2791cceab9894d63ae1780ee054d2c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:34 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:46608
I20260810 22:56:34.261881 32314 tablet_service.cc:1514] Processing DeleteTablet for tablet 2791cceab9894d63ae1780ee054d2c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:34 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56888
I20260810 22:56:34.261961 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet 2791cceab9894d63ae1780ee054d2c62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:34 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57368
I20260810 22:56:34.262261 32728 tablet_replica.cc:330] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: stopping tablet replica
I20260810 22:56:34.262305 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 32078
I20260810 22:56:34.262228 32726 tablet_replica.cc:330] T 2791cceab9894d63ae1780ee054d2c62 P ddbc409f08094fbf94bef416b37bea28: stopping tablet replica
I20260810 22:56:34.264031 32728 raft_consensus.cc:2229] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:34.264467 32728 raft_consensus.cc:2258] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:34.265198 32728 ts_tablet_manager.cc:1905] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:34.265398 32727 tablet_replica.cc:330] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: stopping tablet replica
I20260810 22:56:34.265542 32727 raft_consensus.cc:2229] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:34.265650 32727 raft_consensus.cc:2258] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:34.267503 32728 ts_tablet_manager.cc:1918] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:56:34.267573 32728 log.cc:1196] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/2791cceab9894d63ae1780ee054d2c62
I20260810 22:56:34.267881 32728 ts_tablet_manager.cc:1939] T 2791cceab9894d63ae1780ee054d2c62 P 7e740c72a9e04824820bbc12ceffa197: Deleting consensus metadata
I20260810 22:56:34.268563 31982 catalog_manager.cc:4841] TS 7e740c72a9e04824820bbc12ceffa197 (127.29.78.131:46555): tablet 2791cceab9894d63ae1780ee054d2c62 (table test-table [id=e764817d7ba644048b7f24a3cae082b8]) successfully deleted
I20260810 22:56:34.270210 32727 ts_tablet_manager.cc:1905] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:34.272481 32727 ts_tablet_manager.cc:1918] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 22:56:34.272552 32727 log.cc:1196] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/2791cceab9894d63ae1780ee054d2c62
I20260810 22:56:34.272820 32727 ts_tablet_manager.cc:1939] T 2791cceab9894d63ae1780ee054d2c62 P 049c0e83e7f24a2ea4e098dbc700bc56: Deleting consensus metadata
I20260810 22:56:34.273471 31979 catalog_manager.cc:4841] TS 049c0e83e7f24a2ea4e098dbc700bc56 (127.29.78.130:40405): tablet 2791cceab9894d63ae1780ee054d2c62 (table test-table [id=e764817d7ba644048b7f24a3cae082b8]) successfully deleted
W20260810 22:56:34.274439 31981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:40943 (error 108)
I20260810 22:56:34.274704 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 32239
I20260810 22:56:34.277107 32730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:34.276479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40943 (local address 127.0.0.1:46636)
0810 22:56:34.276956 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:34.277027 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":279,"threads_started":1}
W20260810 22:56:34.277276 31981 catalog_manager.cc:4584] TS ddbc409f08094fbf94bef416b37bea28 (127.29.78.129:40943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2791cceab9894d63ae1780ee054d2c62: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40943: connect: Connection refused (error 111)
I20260810 22:56:34.282351 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 32401
I20260810 22:56:34.291040 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 31961
2026-08-10T22:56:34Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6199 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:34 dist-test-slave-6nmv krb5kdc[32733](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:34 dist-test-slave-6nmv krb5kdc[32733](info): set up 2 sockets
Aug 10 22:56:34 dist-test-slave-6nmv krb5kdc[32733](info): commencing operation
krb5kdc: starting...
W20260810 22:56:35.352346 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:35 dist-test-slave-6nmv krb5kdc[32733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:35.470909 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36145
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36145
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:35.576020 32749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:35.576272 32749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:35.576309 32749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:35.580032 32749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:35.580106 32749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:35.580127 32749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:35.580144 32749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:35.580161 32749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:35.585587 32749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:35.586582 32749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:35.587543 32749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:35.596035 32756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:35.595989 32755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:35.596000 32758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:35.596222 32749 server_base.cc:1034] running on GCE node
I20260810 22:56:35.596764 32749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:35.597870 32749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:35.599046 32749 hybrid_clock.cc:648] HybridClock initialized: now 1786402595599039 us; error 28 us; skew 500 ppm
Aug 10 22:56:35 dist-test-slave-6nmv krb5kdc[32733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402595, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:35.601672 32749 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:35.602520 32749 webserver.cc:460] Webserver started at http://127.29.78.190:41043/ using document root <none> and password file <none>
I20260810 22:56:35.603014 32749 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:35.603096 32749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:35.603302 32749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:35.605273 32749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "123d8df3e52c4f42899c36f95b929f12"
format_stamp: "Formatted at 2026-08-10 22:56:35 on dist-test-slave-6nmv"
server_key: "8de53c830c050e029ab5b4e018e9ff64"
server_key_iv: "a11bda48852eca245dcce2408f19733e"
server_key_version: "encryptionkey@0"
I20260810 22:56:35.605978 32749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "123d8df3e52c4f42899c36f95b929f12"
format_stamp: "Formatted at 2026-08-10 22:56:35 on dist-test-slave-6nmv"
server_key: "8de53c830c050e029ab5b4e018e9ff64"
server_key_iv: "a11bda48852eca245dcce2408f19733e"
server_key_version: "encryptionkey@0"
I20260810 22:56:35.609565 32749 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:35.611914 32767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:35.613106 32749 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:56:35.613237 32749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "123d8df3e52c4f42899c36f95b929f12"
format_stamp: "Formatted at 2026-08-10 22:56:35 on dist-test-slave-6nmv"
server_key: "8de53c830c050e029ab5b4e018e9ff64"
server_key_iv: "a11bda48852eca245dcce2408f19733e"
server_key_version: "encryptionkey@0"
I20260810 22:56:35.613337 32749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:35.625974 32749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:35.628428 32749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:35.628607 32749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:35.635619 32749 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36145
I20260810 22:56:35.635650 357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36145 every 8 connection(s)
I20260810 22:56:35.636451 32749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:35.638932 358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:35.643764 358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12: Bootstrap starting.
I20260810 22:56:35.646173 358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:35.646646 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 32749
I20260810 22:56:35.646744 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:35.646996 30010 external_mini_cluster.cc:1428] Setting key a7cf16a9262f2428b09f9eca32c3d54e
I20260810 22:56:35.646991 358 log.cc:824] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:35.648844 358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12: No bootstrap required, opened a new log
I20260810 22:56:35.652752 358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123d8df3e52c4f42899c36f95b929f12" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36145 } }
I20260810 22:56:35.652961 358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:35.653012 358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123d8df3e52c4f42899c36f95b929f12, State: Initialized, Role: FOLLOWER
I20260810 22:56:35.653431 358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123d8df3e52c4f42899c36f95b929f12" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36145 } }
I20260810 22:56:35.653548 358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:35.653646 358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:35.653765 358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:35.655025 358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123d8df3e52c4f42899c36f95b929f12" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36145 } }
I20260810 22:56:35.655383 358 leader_election.cc:302] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123d8df3e52c4f42899c36f95b929f12; no voters:
Aug 10 22:56:35 dist-test-slave-6nmv krb5kdc[32733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:35.655709 358 leader_election.cc:288] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:35.655764 363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:35.655946 363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [term 1 LEADER]: Becoming Leader. State: Replica: 123d8df3e52c4f42899c36f95b929f12, State: Running, Role: LEADER
I20260810 22:56:35.656363 363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123d8df3e52c4f42899c36f95b929f12" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36145 } }
I20260810 22:56:35.656852 358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:35.658926 366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "123d8df3e52c4f42899c36f95b929f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123d8df3e52c4f42899c36f95b929f12" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36145 } } }
I20260810 22:56:35.659332 366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:35.659420 365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123d8df3e52c4f42899c36f95b929f12. Latest consensus state: current_term: 1 leader_uuid: "123d8df3e52c4f42899c36f95b929f12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123d8df3e52c4f42899c36f95b929f12" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36145 } } }
I20260810 22:56:35.659570 365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:35.661384 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:35.648007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.29.78.190:36145)
0810 22:56:35.648343 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:35.648357 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:35.648394 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:56:35.648982 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:35.648998 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:35.649282 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:35.649530 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:35.650462 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:35.651491 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:35.652644 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:35.653169 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:35.653335 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:35.656338 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:35.656343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:35.656356 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:35.656385 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:35.658349 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:35.658859 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:35.658864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:35.658869 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:35.658998 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:35.659302 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:35.659305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:35.659306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:35.659773 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:35.659975 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:35.660282 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 22:56:35.660562 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":114,"threads_started":1}
W20260810 22:56:35.661775 379 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:56:35.661860 379 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:56:35.661955 382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:35.662729 382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:35.667835 382 catalog_manager.cc:1347] Generated new cluster ID: 9261eca8be2e440e89db644d40c27411
I20260810 22:56:35.667891 382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:35.674697 382 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:35.675591 382 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:35.682562 382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 123d8df3e52c4f42899c36f95b929f12: Generated new TSK 0
I20260810 22:56:35.683178 382 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:35.710901 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36145
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:35.818953 388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:35.819170 388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:35.819234 388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:35.822808 388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:35.822851 388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:35.822960 388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:35.827723 388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:35.828644 388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:35.829555 388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:35.838441 395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:35.838575 394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:35.838656 397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:35.838794 388 server_base.cc:1034] running on GCE node
I20260810 22:56:35.839088 388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:35.839635 388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:35.840787 388 hybrid_clock.cc:648] HybridClock initialized: now 1786402595840756 us; error 45 us; skew 500 ppm
Aug 10 22:56:35 dist-test-slave-6nmv krb5kdc[32733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402595, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:35.843408 388 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:35.844275 388 webserver.cc:460] Webserver started at http://127.29.78.129:34057/ using document root <none> and password file <none>
I20260810 22:56:35.844812 388 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:35.844903 388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:35.845131 388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:35.847090 388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "639af740bc88415eb10470958167bec0"
format_stamp: "Formatted at 2026-08-10 22:56:35 on dist-test-slave-6nmv"
server_key: "dee165dd49228ebb60bdca61ed282ac7"
server_key_iv: "7f9af0b1d8ed30025c3ec84102f82e52"
server_key_version: "encryptionkey@0"
I20260810 22:56:35.847645 388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "639af740bc88415eb10470958167bec0"
format_stamp: "Formatted at 2026-08-10 22:56:35 on dist-test-slave-6nmv"
server_key: "dee165dd49228ebb60bdca61ed282ac7"
server_key_iv: "7f9af0b1d8ed30025c3ec84102f82e52"
server_key_version: "encryptionkey@0"
I20260810 22:56:35.852186 388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 22:56:35.854595 405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:35.855752 388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:35.855882 388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "639af740bc88415eb10470958167bec0"
format_stamp: "Formatted at 2026-08-10 22:56:35 on dist-test-slave-6nmv"
server_key: "dee165dd49228ebb60bdca61ed282ac7"
server_key_iv: "7f9af0b1d8ed30025c3ec84102f82e52"
server_key_version: "encryptionkey@0"
I20260810 22:56:35.855986 388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:35.872056 388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:35.874706 388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:35.874900 388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:35.875469 388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:35.876223 388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:35.876286 388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:35.876345 388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:35.876386 388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:35.885267 388 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39899
I20260810 22:56:35.885351 533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39899 every 8 connection(s)
I20260810 22:56:35.886183 388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:35.887008 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 388
I20260810 22:56:35.887118 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:35.887389 30010 external_mini_cluster.cc:1428] Setting key f4cb4ff76308a4914a97e04bc70200ed
Aug 10 22:56:35 dist-test-slave-6nmv krb5kdc[32733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402595, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:35.901394 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:35.887958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54183 (local address 127.29.78.190:36145)
0810 22:56:35.888100 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:35.888105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:35.889004 (+ 899us) server_negotiation.cc:407] Connection header received
0810 22:56:35.891026 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:35.891031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:35.891090 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:35.891175 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:35.892423 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:35.892977 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:35.894770 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:35.895047 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:35.895131 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:35.897952 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:35.897955 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:35.897957 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:35.897979 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:35.899295 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:35.899807 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:35.899813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:35.899817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:35.899910 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:35.900230 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:35.900233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:35.900235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:35.900434 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:35.900539 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:35.901091 (+ 552us) server_negotiation.cc:299] Negotiation successful
0810 22:56:35.901210 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 22:56:35.902345 538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:35.888361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36145 (local address 127.29.78.129:54183)
0810 22:56:35.888856 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:35.888906 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:35.890795 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:35.891331 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:35.891337 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:35.891688 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:35.892266 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:35.892284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:35.893131 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:35.893137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:35.894621 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:35.894636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:35.895241 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:35.895247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:35.895510 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:35.896165 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:35.896191 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:35.897731 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:35.899444 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:35.899449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:35.899461 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:35.899699 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:35.900001 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:35.900004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:35.900006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:35.900135 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:35.900539 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:35.900545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:35.900779 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 22:56:35.901036 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 22:56:35.901323 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":132,"threads_started":1}
I20260810 22:56:35.903630 535 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36145
I20260810 22:56:35.903895 535 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:35.904538 535 heartbeater.cc:511] Master 127.29.78.190:36145 requested a full tablet report, sending...
I20260810 22:56:35.906313 316 ts_manager.cc:194] Registered new tserver with Master: 639af740bc88415eb10470958167bec0 (127.29.78.129:39899)
I20260810 22:56:35.907469 316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:54183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:35.918871 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36145
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:36.026526 541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:36.026809 541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:36.026885 541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:36.030606 541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:36.030679 541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:36.030766 541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:36.036049 541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:36.037030 541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:36.038117 541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:36.047350 547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:36.047380 546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:36.047380 549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:36.047451 541 server_base.cc:1034] running on GCE node
I20260810 22:56:36.048096 541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:36.048693 541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:36.049890 541 hybrid_clock.cc:648] HybridClock initialized: now 1786402596049880 us; error 32 us; skew 500 ppm
Aug 10 22:56:36 dist-test-slave-6nmv krb5kdc[32733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402596, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:36.052613 541 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:36.053476 541 webserver.cc:460] Webserver started at http://127.29.78.130:37059/ using document root <none> and password file <none>
I20260810 22:56:36.054093 541 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:36.054188 541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:36.054414 541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:36.056366 541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "0a2f586793de413ba21fc3063bf62e8c"
format_stamp: "Formatted at 2026-08-10 22:56:36 on dist-test-slave-6nmv"
server_key: "c9db7393fe0e15ad52a934a70b0c6d5c"
server_key_iv: "fd9cc8aef8eaf3268eb561fda9a13cff"
server_key_version: "encryptionkey@0"
I20260810 22:56:36.056954 541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "0a2f586793de413ba21fc3063bf62e8c"
format_stamp: "Formatted at 2026-08-10 22:56:36 on dist-test-slave-6nmv"
server_key: "c9db7393fe0e15ad52a934a70b0c6d5c"
server_key_iv: "fd9cc8aef8eaf3268eb561fda9a13cff"
server_key_version: "encryptionkey@0"
I20260810 22:56:36.060706 541 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:56:36.063659 557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:36.064819 541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:56:36.064945 541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "0a2f586793de413ba21fc3063bf62e8c"
format_stamp: "Formatted at 2026-08-10 22:56:36 on dist-test-slave-6nmv"
server_key: "c9db7393fe0e15ad52a934a70b0c6d5c"
server_key_iv: "fd9cc8aef8eaf3268eb561fda9a13cff"
server_key_version: "encryptionkey@0"
I20260810 22:56:36.065064 541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:36.082509 541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:36.085268 541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:36.085484 541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:36.086032 541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:36.086807 541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:36.086879 541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:36.086949 541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:36.086992 541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:36.096068 541 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:33823
I20260810 22:56:36.096099 681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:33823 every 8 connection(s)
I20260810 22:56:36.096884 541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:36.105211 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 541
I20260810 22:56:36.105304 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:36.105593 30010 external_mini_cluster.cc:1428] Setting key e3f159b9d4243f8778831e8d21264776
Aug 10 22:56:36 dist-test-slave-6nmv krb5kdc[32733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402596, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:36.110123 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:36.098687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:59023 (local address 127.29.78.190:36145)
0810 22:56:36.098828 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:36.098832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:36.099587 (+ 755us) server_negotiation.cc:407] Connection header received
0810 22:56:36.100900 (+ 1313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:36.100903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:36.100959 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:36.101046 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:36.102223 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.102740 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.103826 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.104057 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.104171 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:36.106667 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:36.106670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:36.106673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:36.106699 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:36.108135 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:36.108615 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:36.108618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:36.108620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:36.108702 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:36.109050 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:36.109055 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:36.109057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:36.109246 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:36.109369 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:36.109845 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 22:56:36.109967 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:56:36.110891 684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:36.099032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36145 (local address 127.29.78.130:59023)
0810 22:56:36.099451 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:36.099490 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:36.100701 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:36.101221 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:36.101227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:36.101559 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:36.102081 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:36.102094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.102887 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.102890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:36.103702 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:36.103711 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.104230 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.104235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:36.104395 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:36.104916 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:36.104931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:36.106513 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:36.108250 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:36.108256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:36.108269 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:36.108514 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:36.108792 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:36.108795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:36.108797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:36.108955 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:36.109369 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:36.109374 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:36.109603 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 22:56:36.109781 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 22:56:36.110113 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":110,"threads_started":1}
I20260810 22:56:36.113046 682 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36145
I20260810 22:56:36.113272 682 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:36.113714 682 heartbeater.cc:511] Master 127.29.78.190:36145 requested a full tablet report, sending...
I20260810 22:56:36.114804 316 ts_manager.cc:194] Registered new tserver with Master: 0a2f586793de413ba21fc3063bf62e8c (127.29.78.130:33823)
I20260810 22:56:36.115655 316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:59023
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:36.133131 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36145
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:36.237020 687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:36.237224 687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:36.237262 687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:36.240880 687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:36.240943 687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:36.241021 687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:36.245654 687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36145
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:36.246538 687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:36.247426 687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:36.256469 692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:36.256444 696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:36.256506 693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:36.256747 687 server_base.cc:1034] running on GCE node
I20260810 22:56:36.257211 687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:36.257925 687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:36.259155 687 hybrid_clock.cc:648] HybridClock initialized: now 1786402596259120 us; error 48 us; skew 500 ppm
Aug 10 22:56:36 dist-test-slave-6nmv krb5kdc[32733](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402596, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:36.261775 687 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:36.262641 687 webserver.cc:460] Webserver started at http://127.29.78.131:43867/ using document root <none> and password file <none>
I20260810 22:56:36.263157 687 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:36.263242 687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:36.263451 687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:36.265317 687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "91883adc6a524e41a5f4db63f1724a13"
format_stamp: "Formatted at 2026-08-10 22:56:36 on dist-test-slave-6nmv"
server_key: "22ff3aa285e4d802262e335d1e971e50"
server_key_iv: "a85e2bd1971b370e62249a44223cd9f7"
server_key_version: "encryptionkey@0"
I20260810 22:56:36.265929 687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "91883adc6a524e41a5f4db63f1724a13"
format_stamp: "Formatted at 2026-08-10 22:56:36 on dist-test-slave-6nmv"
server_key: "22ff3aa285e4d802262e335d1e971e50"
server_key_iv: "a85e2bd1971b370e62249a44223cd9f7"
server_key_version: "encryptionkey@0"
I20260810 22:56:36.269330 687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:56:36.271658 703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:36.272742 687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:36.272878 687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "91883adc6a524e41a5f4db63f1724a13"
format_stamp: "Formatted at 2026-08-10 22:56:36 on dist-test-slave-6nmv"
server_key: "22ff3aa285e4d802262e335d1e971e50"
server_key_iv: "a85e2bd1971b370e62249a44223cd9f7"
server_key_version: "encryptionkey@0"
I20260810 22:56:36.272974 687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:36.290422 687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:36.292964 687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:36.293147 687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:36.293640 687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:36.294458 687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:36.294530 687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:36.294593 687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:36.294637 687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:36.302981 687 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:35571
I20260810 22:56:36.303025 834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:35571 every 8 connection(s)
I20260810 22:56:36.303807 687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:36.308938 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 687
I20260810 22:56:36.309067 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:36.309345 30010 external_mini_cluster.cc:1428] Setting key 08d51088afcef2280c04197734bd347a
Aug 10 22:56:36 dist-test-slave-6nmv krb5kdc[32733](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402596, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:36.317770 362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:36.305475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46177 (local address 127.29.78.190:36145)
0810 22:56:36.305607 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:36.305611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:36.306419 (+ 808us) server_negotiation.cc:407] Connection header received
0810 22:56:36.307756 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:36.307759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:36.307816 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:36.307892 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:36.309089 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.309710 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.311052 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.311358 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.311459 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:36.313781 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:36.313784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:36.313786 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:36.313846 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:36.315588 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:36.316193 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:36.316195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:36.316197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:36.316292 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:36.316569 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:36.316571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:36.316572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:36.316801 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:36.316909 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:36.317446 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 22:56:36.317605 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:56:36.318478 837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:36.305846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36145 (local address 127.29.78.131:46177)
0810 22:56:36.306288 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:36.306326 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:36.307535 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:36.308051 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:36.308057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:36.308373 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:36.308904 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:36.308927 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.309864 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.309867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:36.310937 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:36.310946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:36.311503 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:36.311512 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:36.311678 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:36.312250 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:36.312266 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:36.313631 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:36.315717 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:36.315722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:36.315736 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:36.316087 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:36.316351 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:36.316353 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:36.316354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:36.316491 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:36.316923 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:36.316931 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:36.317174 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 22:56:36.317391 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 22:56:36.317698 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":103,"threads_started":1}
I20260810 22:56:36.319479 835 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36145
I20260810 22:56:36.319710 835 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:36.320153 835 heartbeater.cc:511] Master 127.29.78.190:36145 requested a full tablet report, sending...
I20260810 22:56:36.321090 316 ts_manager.cc:194] Registered new tserver with Master: 91883adc6a524e41a5f4db63f1724a13 (127.29.78.131:35571)
I20260810 22:56:36.321602 316 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:46177
I20260810 22:56:36.323020 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:56:36.337231 362 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:36.336019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41010 (local address 127.29.78.190:36145)
0810 22:56:36.336159 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:36.336163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:36.336305 (+ 142us) server_negotiation.cc:407] Connection header received
0810 22:56:36.336396 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:36.336398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:36.336435 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:36.336503 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:36.337111 (+ 608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41010: BlockingRecv error: recv got EOF from 127.0.0.1:41010 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:56:36.338716 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 388
I20260810 22:56:36.345845 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 541
I20260810 22:56:36.356678 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 687
I20260810 22:56:36.362973 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 32749
2026-08-10T22:56:36Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2070 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:36 dist-test-slave-6nmv krb5kdc[848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:36 dist-test-slave-6nmv krb5kdc[848](info): set up 2 sockets
Aug 10 22:56:36 dist-test-slave-6nmv krb5kdc[848](info): commencing operation
krb5kdc: starting...
W20260810 22:56:37.408138 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:37 dist-test-slave-6nmv krb5kdc[848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:37.526490 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42761
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:37.631672 865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:37.631913 865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:37.631950 865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:37.635699 865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:37.635735 865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:37.635752 865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:37.635766 865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:37.635778 865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:37.640029 865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:37.640911 865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:37.641819 865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:37.649784 875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:37.649984 865 server_base.cc:1034] running on GCE node
W20260810 22:56:37.649845 873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:37.649797 872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:37.650660 865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:37.651664 865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:37.652842 865 hybrid_clock.cc:648] HybridClock initialized: now 1786402597652817 us; error 40 us; skew 500 ppm
Aug 10 22:56:37 dist-test-slave-6nmv krb5kdc[848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402597, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:37.655637 865 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:37.656548 865 webserver.cc:460] Webserver started at http://127.29.78.190:45997/ using document root <none> and password file <none>
I20260810 22:56:37.657099 865 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:37.657161 865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:37.657410 865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:37.659458 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "f8b31fc36cc040cb879e6b1fdead94a4"
format_stamp: "Formatted at 2026-08-10 22:56:37 on dist-test-slave-6nmv"
server_key: "1e35ce301f24be56e33acba7d0f1876b"
server_key_iv: "df907ce44c7d0a860e5c1edf2da0aa60"
server_key_version: "encryptionkey@0"
I20260810 22:56:37.660027 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "f8b31fc36cc040cb879e6b1fdead94a4"
format_stamp: "Formatted at 2026-08-10 22:56:37 on dist-test-slave-6nmv"
server_key: "1e35ce301f24be56e33acba7d0f1876b"
server_key_iv: "df907ce44c7d0a860e5c1edf2da0aa60"
server_key_version: "encryptionkey@0"
I20260810 22:56:37.663722 865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 22:56:37.666134 885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:37.667315 865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:56:37.667454 865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "f8b31fc36cc040cb879e6b1fdead94a4"
format_stamp: "Formatted at 2026-08-10 22:56:37 on dist-test-slave-6nmv"
server_key: "1e35ce301f24be56e33acba7d0f1876b"
server_key_iv: "df907ce44c7d0a860e5c1edf2da0aa60"
server_key_version: "encryptionkey@0"
I20260810 22:56:37.667551 865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:37.680318 865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:37.683045 865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:37.683182 865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:37.689779 865 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:42761
I20260810 22:56:37.690044 939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:42761 every 8 connection(s)
I20260810 22:56:37.690630 865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:37.691568 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 865
I20260810 22:56:37.691723 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:37.692010 30010 external_mini_cluster.cc:1428] Setting key 341fe41a350e947cc910e18dfadbad41
I20260810 22:56:37.693733 940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:37.699916 940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4: Bootstrap starting.
Aug 10 22:56:37 dist-test-slave-6nmv krb5kdc[848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:37.702711 940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:37.703691 940 log.cc:824] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:37.705611 940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4: No bootstrap required, opened a new log
I20260810 22:56:37.706097 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:37.693229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.29.78.190:42761)
0810 22:56:37.693654 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:37.693669 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:37.693703 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:56:37.694399 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:37.694414 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:37.694729 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:37.695002 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:37.695971 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:37.696846 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:37.697936 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:37.698357 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:37.698461 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:37.701262 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:37.701266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:37.701277 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:37.701301 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:37.702972 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:37.703412 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:37.703417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:37.703423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:37.703536 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:37.703803 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:37.703807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:37.703808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:37.704184 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:37.704425 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:37.704758 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 22:56:37.705200 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":176,"threads_started":1}
I20260810 22:56:37.709692 940 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42761 } }
I20260810 22:56:37.709940 940 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:37.710021 940 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8b31fc36cc040cb879e6b1fdead94a4, State: Initialized, Role: FOLLOWER
I20260810 22:56:37.710433 940 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42761 } }
I20260810 22:56:37.710551 940 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:37.710631 940 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:37.710713 940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:37.711730 940 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42761 } }
I20260810 22:56:37.712054 940 leader_election.cc:302] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8b31fc36cc040cb879e6b1fdead94a4; no voters:
I20260810 22:56:37.712352 940 leader_election.cc:288] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:37.712481 947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:37.712738 947 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [term 1 LEADER]: Becoming Leader. State: Replica: f8b31fc36cc040cb879e6b1fdead94a4, State: Running, Role: LEADER
I20260810 22:56:37.713073 947 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42761 } }
I20260810 22:56:37.713585 940 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:37.714255 947 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8b31fc36cc040cb879e6b1fdead94a4. Latest consensus state: current_term: 1 leader_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42761 } } }
I20260810 22:56:37.714255 948 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8b31fc36cc040cb879e6b1fdead94a4" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42761 } } }
I20260810 22:56:37.714540 947 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:37.714582 948 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:37.714984 958 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:37.716149 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:56:37.718145 958 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:37.724509 958 catalog_manager.cc:1347] Generated new cluster ID: 77d3f5f3589c411fabf1b2f6119e3aee
I20260810 22:56:37.724573 958 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:37.743129 958 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:37.744366 958 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:37.746431 944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:37.741095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.29.78.190:42761)
0810 22:56:37.741224 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:37.741228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:37.741421 (+ 193us) server_negotiation.cc:407] Connection header received
0810 22:56:37.741642 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:37.741646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:37.741704 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:37.741847 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:37.742492 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:37.743163 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:37.744086 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:37.744351 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:37.744467 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:37.745398 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:37.745402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:37.745405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:37.745933 (+ 528us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:37.745961 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:37.745977 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:37.746074 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:37.746184 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 22:56:37.746270 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:56:37.754573 958 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8b31fc36cc040cb879e6b1fdead94a4: Generated new TSK 0
I20260810 22:56:37.755249 958 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:56:37.763454 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 865
2026-08-10T22:56:37Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1399 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:37 dist-test-slave-6nmv krb5kdc[978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:37 dist-test-slave-6nmv krb5kdc[978](info): set up 2 sockets
Aug 10 22:56:37 dist-test-slave-6nmv krb5kdc[978](info): commencing operation
krb5kdc: starting...
W20260810 22:56:38.817979 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:38 dist-test-slave-6nmv krb5kdc[978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:38.930490 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34669
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:39.036664 991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:39.036924 991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:39.037012 991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:39.040714 991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:39.040793 991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:39.040830 991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:39.040897 991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:39.040952 991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:39.046064 991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:39.047061 991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:39.048018 991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:39.056267 998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.056280 997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.056272 1000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:39.056453 991 server_base.cc:1034] running on GCE node
I20260810 22:56:39.057113 991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:39.058179 991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:39.059424 991 hybrid_clock.cc:648] HybridClock initialized: now 1786402599059407 us; error 35 us; skew 500 ppm
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:39.062292 991 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:39.063166 991 webserver.cc:460] Webserver started at http://127.29.78.190:36931/ using document root <none> and password file <none>
I20260810 22:56:39.063701 991 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:39.063795 991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:39.064024 991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:39.066185 991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "8e0a92f618e14ff0a722e5a215731359"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "3c5b3108d03acaace1217ecc529b97c2"
server_key_iv: "ca65468b36b136c0524b8182ac4e5b22"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.066767 991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "8e0a92f618e14ff0a722e5a215731359"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "3c5b3108d03acaace1217ecc529b97c2"
server_key_iv: "ca65468b36b136c0524b8182ac4e5b22"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.070461 991 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:56:39.072748 1008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.073932 991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:39.074074 991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "8e0a92f618e14ff0a722e5a215731359"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "3c5b3108d03acaace1217ecc529b97c2"
server_key_iv: "ca65468b36b136c0524b8182ac4e5b22"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.074187 991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:39.096959 991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:39.100201 991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:39.100410 991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:39.107506 991 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:34669
I20260810 22:56:39.107798 1067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:34669 every 8 connection(s)
I20260810 22:56:39.108448 991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:39.111141 1068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:39.116304 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 991
I20260810 22:56:39.116487 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:39.116564 1068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359: Bootstrap starting.
I20260810 22:56:39.116809 30010 external_mini_cluster.cc:1428] Setting key 16711b22fa10e086cb0b54e678b1bde8
I20260810 22:56:39.119282 1068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:39.120107 1068 log.cc:824] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:39.122354 1068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359: No bootstrap required, opened a new log
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:39.126628 1068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0a92f618e14ff0a722e5a215731359" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34669 } }
I20260810 22:56:39.126926 1068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:39.127018 1068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0a92f618e14ff0a722e5a215731359, State: Initialized, Role: FOLLOWER
I20260810 22:56:39.127501 1068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0a92f618e14ff0a722e5a215731359" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34669 } }
I20260810 22:56:39.127632 1068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:39.127731 1068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:39.127882 1068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:39.128800 1068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0a92f618e14ff0a722e5a215731359" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34669 } }
I20260810 22:56:39.129149 1068 leader_election.cc:302] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e0a92f618e14ff0a722e5a215731359; no voters:
I20260810 22:56:39.129472 1068 leader_election.cc:288] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:39.129575 1073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:39.129807 1073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [term 1 LEADER]: Becoming Leader. State: Replica: 8e0a92f618e14ff0a722e5a215731359, State: Running, Role: LEADER
I20260810 22:56:39.130187 1073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0a92f618e14ff0a722e5a215731359" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34669 } }
I20260810 22:56:39.130803 1068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:39.131719 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e0a92f618e14ff0a722e5a215731359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0a92f618e14ff0a722e5a215731359" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34669 } } }
I20260810 22:56:39.131856 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:39.132079 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e0a92f618e14ff0a722e5a215731359. Latest consensus state: current_term: 1 leader_uuid: "8e0a92f618e14ff0a722e5a215731359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e0a92f618e14ff0a722e5a215731359" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34669 } } }
I20260810 22:56:39.132210 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:39.132705 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.118257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53398 (local address 127.29.78.190:34669)
0810 22:56:39.118694 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:39.118711 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:39.118750 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:56:39.119540 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:39.119562 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:39.119951 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:39.120293 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:39.121233 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.122685 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.123520 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.124196 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.124351 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.127215 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:39.127220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:39.127231 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:39.127256 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:39.129187 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.129746 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.129755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.129764 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.129951 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.130240 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.130244 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.130245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.130732 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:39.130955 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:39.131333 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 22:56:39.131695 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":146,"threads_started":1}
I20260810 22:56:39.133075 1078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:39.136250 1078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:39.141706 1078 catalog_manager.cc:1347] Generated new cluster ID: e61b3a842a6e4aae8dba982a8bf29c63
I20260810 22:56:39.141788 1078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:39.151824 1078 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:39.152716 1078 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:39.157729 1078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e0a92f618e14ff0a722e5a215731359: Generated new TSK 0
I20260810 22:56:39.158385 1078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:39.192588 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34669
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:39.304709 1096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:39.304951 1096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:39.304996 1096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:39.308621 1096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:39.308676 1096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:39.308727 1096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:39.313349 1096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:39.314251 1096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:39.315235 1096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:56:39.323966 1096 server_base.cc:1034] running on GCE node
W20260810 22:56:39.323907 1104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.323930 1101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.323907 1102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:39.324604 1096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:39.325284 1096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:39.326480 1096 hybrid_clock.cc:648] HybridClock initialized: now 1786402599326465 us; error 35 us; skew 500 ppm
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:39.329375 1096 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:39.330322 1096 webserver.cc:460] Webserver started at http://127.29.78.129:46285/ using document root <none> and password file <none>
I20260810 22:56:39.330857 1096 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:39.330951 1096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:39.331156 1096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:39.333871 1096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "a96884a6c85642bc9b1eb2e4333f348e"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "6b71846745d9952382fbde53748407bd"
server_key_iv: "9383a514228be45380d34c1e4d9e4bf3"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.334450 1096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "a96884a6c85642bc9b1eb2e4333f348e"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "6b71846745d9952382fbde53748407bd"
server_key_iv: "9383a514228be45380d34c1e4d9e4bf3"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.338189 1096 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:56:39.340525 1115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.341632 1096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:39.341763 1096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "a96884a6c85642bc9b1eb2e4333f348e"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "6b71846745d9952382fbde53748407bd"
server_key_iv: "9383a514228be45380d34c1e4d9e4bf3"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.341912 1096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:39.359692 1096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:39.362200 1096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:39.362375 1096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:39.362980 1096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:39.363781 1096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:39.363855 1096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.363925 1096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:39.363978 1096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.372635 1096 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:36047
I20260810 22:56:39.372676 1241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:36047 every 8 connection(s)
I20260810 22:56:39.373438 1096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:39.379385 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 1096
I20260810 22:56:39.379493 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:39.379756 30010 external_mini_cluster.cc:1428] Setting key 415bae4d6ff3bf09a8d1f4795eae2d97
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:39.387437 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.375162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:51993 (local address 127.29.78.190:34669)
0810 22:56:39.375294 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:39.375297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:39.376050 (+ 753us) server_negotiation.cc:407] Connection header received
0810 22:56:39.377457 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:39.377461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:39.377545 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:39.377634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:39.378747 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.379260 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.380524 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.380761 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.380879 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.383613 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:39.383618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:39.383622 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:39.383651 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:39.385327 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.385864 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.385870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.385875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.385969 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.386347 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.386350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.386352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.386565 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:39.386674 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:39.387076 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 22:56:39.387213 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:56:39.387949 1245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.375435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34669 (local address 127.29.78.129:51993)
0810 22:56:39.375894 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:39.375934 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:39.377241 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:39.377792 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:39.377798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:39.378137 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:39.378595 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:39.378607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.379390 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.379393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:39.380406 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:39.380416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.380909 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.380913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:39.381115 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.381673 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:39.381696 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:39.383418 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:39.385442 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:39.385446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:39.385457 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:39.385711 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:39.386083 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:39.386087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:39.386088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:39.386248 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:39.386669 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:39.386673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:39.386864 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 22:56:39.386992 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 22:56:39.387197 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":113,"threads_started":1}
I20260810 22:56:39.389118 1242 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34669
I20260810 22:56:39.389348 1242 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:39.389784 1242 heartbeater.cc:511] Master 127.29.78.190:34669 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:39.391292 1027 ts_manager.cc:194] Registered new tserver with Master: a96884a6c85642bc9b1eb2e4333f348e (127.29.78.129:36047)
I20260810 22:56:39.392390 1027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:51993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:39.408183 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34669
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:39.510500 1248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:39.510735 1248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:39.510788 1248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:39.514631 1248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:39.514678 1248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:39.514732 1248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:39.519129 1248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:39.519955 1248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:39.520778 1248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:39.529208 1257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.529227 1254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.529218 1255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:39.529412 1248 server_base.cc:1034] running on GCE node
I20260810 22:56:39.529850 1248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:39.530360 1248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:39.531531 1248 hybrid_clock.cc:648] HybridClock initialized: now 1786402599531500 us; error 53 us; skew 500 ppm
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:39.534229 1248 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:39.535109 1248 webserver.cc:460] Webserver started at http://127.29.78.130:34187/ using document root <none> and password file <none>
I20260810 22:56:39.535643 1248 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:39.535750 1248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:39.535950 1248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:39.537779 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "69d6d445f2a64adfa9c630289dedbea5"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "6a1e58bb35914dfb46c6db368f8262b5"
server_key_iv: "0e72a2bc2157029f40579963aa7ab749"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.538390 1248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "69d6d445f2a64adfa9c630289dedbea5"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "6a1e58bb35914dfb46c6db368f8262b5"
server_key_iv: "0e72a2bc2157029f40579963aa7ab749"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.541774 1248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:56:39.543943 1267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.545158 1248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:39.545295 1248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "69d6d445f2a64adfa9c630289dedbea5"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "6a1e58bb35914dfb46c6db368f8262b5"
server_key_iv: "0e72a2bc2157029f40579963aa7ab749"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.545405 1248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:39.556582 1248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:39.559087 1248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:39.559278 1248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:39.559801 1248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:39.560525 1248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:39.560595 1248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.560658 1248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:39.560704 1248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.569069 1248 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:36357
I20260810 22:56:39.569103 1395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:36357 every 8 connection(s)
I20260810 22:56:39.569875 1248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:39.573882 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 1248
I20260810 22:56:39.574021 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:39.574323 30010 external_mini_cluster.cc:1428] Setting key 403472911fbb67d16cecf11ca5a8489f
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:39.583504 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.571407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:37969 (local address 127.29.78.190:34669)
0810 22:56:39.571526 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:39.571529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:39.572292 (+ 763us) server_negotiation.cc:407] Connection header received
0810 22:56:39.573923 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:39.573929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:39.574015 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:39.574103 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:39.575333 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.575977 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.577035 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.577394 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.577509 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.579962 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:39.579968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:39.579973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:39.580006 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:39.581516 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.582023 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.582028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.582031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.582120 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.582415 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.582420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.582422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.582630 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:39.582731 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:39.583139 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 22:56:39.583290 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:56:39.584044 1399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.571743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34669 (local address 127.29.78.130:37969)
0810 22:56:39.572155 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:39.572194 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:39.573463 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:39.574260 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:39.574266 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:39.574691 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:39.575199 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:39.575212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.576106 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.576109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:39.576916 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:39.576925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.577551 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.577556 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:39.577691 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.578367 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:39.578381 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:39.579785 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:39.581621 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:39.581624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:39.581634 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:39.581932 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:39.582212 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:39.582214 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:39.582215 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:39.582327 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:39.582736 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:39.582739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:39.582922 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 22:56:39.583048 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 22:56:39.583279 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:56:39.585170 1396 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34669
I20260810 22:56:39.585405 1396 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:39.585917 1396 heartbeater.cc:511] Master 127.29.78.190:34669 requested a full tablet report, sending...
I20260810 22:56:39.586918 1027 ts_manager.cc:194] Registered new tserver with Master: 69d6d445f2a64adfa9c630289dedbea5 (127.29.78.130:36357)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260810 22:56:39.587666 1027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:37969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:39.600621 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34669
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:39.704952 1402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:39.705194 1402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:39.705236 1402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:39.709074 1402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:39.709136 1402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:39.709192 1402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:39.714427 1402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:39.715474 1402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:39.716514 1402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:39.725828 1408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.725767 1410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:39.725908 1407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:39.726508 1402 server_base.cc:1034] running on GCE node
I20260810 22:56:39.726886 1402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:39.727474 1402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:39.728670 1402 hybrid_clock.cc:648] HybridClock initialized: now 1786402599728645 us; error 42 us; skew 500 ppm
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:39.731441 1402 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:39.732453 1402 webserver.cc:460] Webserver started at http://127.29.78.131:41633/ using document root <none> and password file <none>
I20260810 22:56:39.732993 1402 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:39.733052 1402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:39.733240 1402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:39.735179 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "cc2e77b343c5492da0c43a3eab8786b3"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "435d3bacbdbd4a94e29f1e3e7c81ae09"
server_key_iv: "c88a01e27c72ab8a06df16e9f4f0c5cc"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.735762 1402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "cc2e77b343c5492da0c43a3eab8786b3"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "435d3bacbdbd4a94e29f1e3e7c81ae09"
server_key_iv: "c88a01e27c72ab8a06df16e9f4f0c5cc"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.739378 1402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:56:39.741520 1419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.742668 1402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:56:39.742780 1402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "cc2e77b343c5492da0c43a3eab8786b3"
format_stamp: "Formatted at 2026-08-10 22:56:39 on dist-test-slave-6nmv"
server_key: "435d3bacbdbd4a94e29f1e3e7c81ae09"
server_key_iv: "c88a01e27c72ab8a06df16e9f4f0c5cc"
server_key_version: "encryptionkey@0"
I20260810 22:56:39.742872 1402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:39.767078 1402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:39.770378 1402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:39.770582 1402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:39.771189 1402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:39.772018 1402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:39.772065 1402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.772132 1402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:39.772173 1402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:39.781153 1402 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:40039
I20260810 22:56:39.781183 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:40039 every 8 connection(s)
I20260810 22:56:39.781989 1402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:39.787114 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 1402
I20260810 22:56:39.787238 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:39.787487 30010 external_mini_cluster.cc:1428] Setting key 69771186979760bec8b5341456ab8423
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:39.794361 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.783768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:39449 (local address 127.29.78.190:34669)
0810 22:56:39.783908 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:39.783911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:39.784733 (+ 822us) server_negotiation.cc:407] Connection header received
0810 22:56:39.786111 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:39.786113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:39.786159 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:39.786228 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:39.787316 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.787855 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.788921 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.789138 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.789249 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.791477 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:39.791480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:39.791481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:39.791510 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:39.792722 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.793152 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.793154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.793156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.793224 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.793504 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.793506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.793507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.793657 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:39.793725 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:39.794170 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 22:56:39.794257 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:56:39.795135 1555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.784134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34669 (local address 127.29.78.131:39449)
0810 22:56:39.784581 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:39.784622 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:39.785924 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:39.786372 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:39.786378 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:39.786689 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:39.787138 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:39.787150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.787973 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.787978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:39.788791 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:39.788801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.789284 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.789288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:39.789452 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.790013 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:39.790026 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:39.791343 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:39.792817 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:39.792822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:39.792831 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:39.793076 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:39.793303 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:39.793306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:39.793307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:39.793423 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:39.793734 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:39.793738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:39.793985 (+ 247us) client_negotiation.cc:769] Sending connection context
0810 22:56:39.794128 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 22:56:39.794343 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260810 22:56:39.796151 1553 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34669
I20260810 22:56:39.796356 1553 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:39.796772 1553 heartbeater.cc:511] Master 127.29.78.190:34669 requested a full tablet report, sending...
I20260810 22:56:39.797658 1027 ts_manager.cc:194] Registered new tserver with Master: cc2e77b343c5492da0c43a3eab8786b3 (127.29.78.131:40039)
I20260810 22:56:39.798220 1027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:39449
I20260810 22:56:39.800938 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402599, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:39.827149 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:39.819021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.29.78.190:34669)
0810 22:56:39.819203 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:39.819209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:39.819352 (+ 143us) server_negotiation.cc:407] Connection header received
0810 22:56:39.819530 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:39.819532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:39.819575 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:39.819656 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:39.820338 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.820873 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.821686 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:39.821954 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:39.822064 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:39.824414 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:39.824417 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:39.824419 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:39.824439 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:39.825713 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.826149 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.826151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.826152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.826223 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:39.826440 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:39.826442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:39.826443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:39.826652 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:39.826760 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:39.826947 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:56:39.827036 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 22:56:39.827550 1027 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53404
I20260810 22:56:39.828611 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 1096
I20260810 22:56:39.837057 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 1248
I20260810 22:56:39.850070 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 1402
I20260810 22:56:39.856653 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 991
2026-08-10T22:56:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2095 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[1566](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[1566](info): set up 2 sockets
Aug 10 22:56:39 dist-test-slave-6nmv krb5kdc[1566](info): commencing operation
krb5kdc: starting...
W20260810 22:56:40.906551 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:40 dist-test-slave-6nmv krb5kdc[1566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:41.026041 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35179
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:41.133178 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:41.133445 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:41.133543 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:41.137379 1580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:41.137480 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:41.137516 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:41.137569 1580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:41.137607 1580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:41.142383 1580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:41.143334 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:41.144271 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:41.152882 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:41.152890 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:41.152968 1580 server_base.cc:1034] running on GCE node
W20260810 22:56:41.153148 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:41.153683 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:41.154755 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:41.155941 1580 hybrid_clock.cc:648] HybridClock initialized: now 1786402601155923 us; error 35 us; skew 500 ppm
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:41.158581 1580 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:41.159479 1580 webserver.cc:460] Webserver started at http://127.29.78.190:43121/ using document root <none> and password file <none>
I20260810 22:56:41.160012 1580 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:41.160099 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:41.160339 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:41.162312 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "72eca5042eb74074b659264e4102e2ac"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "bf4aa1c143be207c603b75996ceab3f4"
server_key_iv: "c051637f3ea6fa18b870063f044a2a78"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.162851 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "72eca5042eb74074b659264e4102e2ac"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "bf4aa1c143be207c603b75996ceab3f4"
server_key_iv: "c051637f3ea6fa18b870063f044a2a78"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.166347 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:56:41.168478 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.169636 1580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:41.169795 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "72eca5042eb74074b659264e4102e2ac"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "bf4aa1c143be207c603b75996ceab3f4"
server_key_iv: "c051637f3ea6fa18b870063f044a2a78"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.169948 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:41.195561 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:41.198235 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:41.198436 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:41.205461 1580 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35179
I20260810 22:56:41.205487 1657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35179 every 8 connection(s)
I20260810 22:56:41.206378 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:41.208909 1658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:41.212357 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 1580
I20260810 22:56:41.212484 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:41.212797 30010 external_mini_cluster.cc:1428] Setting key 95608beb69940a564a115fb346c099de
I20260810 22:56:41.214895 1658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac: Bootstrap starting.
I20260810 22:56:41.216969 1658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:41.217687 1658 log.cc:824] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:41.219410 1658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac: No bootstrap required, opened a new log
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402600, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:41.222669 1658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eca5042eb74074b659264e4102e2ac" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35179 } }
I20260810 22:56:41.222903 1658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:41.222995 1658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72eca5042eb74074b659264e4102e2ac, State: Initialized, Role: FOLLOWER
I20260810 22:56:41.223455 1658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eca5042eb74074b659264e4102e2ac" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35179 } }
I20260810 22:56:41.223609 1658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:41.223693 1658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:41.223810 1658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:41.224628 1658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eca5042eb74074b659264e4102e2ac" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35179 } }
I20260810 22:56:41.224980 1658 leader_election.cc:302] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72eca5042eb74074b659264e4102e2ac; no voters:
I20260810 22:56:41.225307 1658 leader_election.cc:288] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:41.225376 1663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:41.225646 1663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [term 1 LEADER]: Becoming Leader. State: Replica: 72eca5042eb74074b659264e4102e2ac, State: Running, Role: LEADER
I20260810 22:56:41.226028 1663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eca5042eb74074b659264e4102e2ac" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35179 } }
I20260810 22:56:41.226630 1658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:41.226707 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.214275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.29.78.190:35179)
0810 22:56:41.214711 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:41.214726 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:41.214761 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:56:41.215427 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:41.215445 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:41.215755 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:41.216085 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:41.216838 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.217920 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.218730 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.219185 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.219353 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.221635 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:41.221639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:41.221651 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:41.221675 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:41.223534 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.224039 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.224043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.224047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.224148 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.224424 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.224427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.224429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.224854 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:41.225059 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:41.225318 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 22:56:41.225726 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":142,"threads_started":1}
I20260810 22:56:41.227144 1663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72eca5042eb74074b659264e4102e2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eca5042eb74074b659264e4102e2ac" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35179 } } }
I20260810 22:56:41.227321 1663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:41.227535 1664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72eca5042eb74074b659264e4102e2ac. Latest consensus state: current_term: 1 leader_uuid: "72eca5042eb74074b659264e4102e2ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72eca5042eb74074b659264e4102e2ac" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35179 } } }
I20260810 22:56:41.227690 1664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac [sys.catalog]: This master's current role is: LEADER
W20260810 22:56:41.231213 1683 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:56:41.231287 1683 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:56:41.231356 1672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:41.232168 1672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:41.237272 1672 catalog_manager.cc:1347] Generated new cluster ID: d1c06faa209c409eb57730f20eea45f5
I20260810 22:56:41.237337 1672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:41.256421 1672 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:41.257295 1672 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:41.263481 1672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72eca5042eb74074b659264e4102e2ac: Generated new TSK 0
I20260810 22:56:41.264072 1672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:41.294461 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35179
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:41.403757 1689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:41.403975 1689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:41.404031 1689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:41.407585 1689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:41.407626 1689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:41.407672 1689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:41.412189 1689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:41.413093 1689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:41.413987 1689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:41.422693 1695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:41.422701 1694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:41.422789 1697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:41.422958 1689 server_base.cc:1034] running on GCE node
I20260810 22:56:41.423451 1689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:41.424043 1689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:41.425199 1689 hybrid_clock.cc:648] HybridClock initialized: now 1786402601425186 us; error 24 us; skew 500 ppm
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:41.428128 1689 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:41.429021 1689 webserver.cc:460] Webserver started at http://127.29.78.129:44227/ using document root <none> and password file <none>
I20260810 22:56:41.429548 1689 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:41.429605 1689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:41.429802 1689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:41.431690 1689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "45f9ddbb798d4a1d9f3fee07aa3f441f"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "9259830970158e341ed80d4a76cea978"
server_key_iv: "6085ea32b4a8aaf4b31a8b3547eb6078"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.432233 1689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "45f9ddbb798d4a1d9f3fee07aa3f441f"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "9259830970158e341ed80d4a76cea978"
server_key_iv: "6085ea32b4a8aaf4b31a8b3547eb6078"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.435729 1689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:41.437896 1705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.438887 1689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:41.439016 1689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "45f9ddbb798d4a1d9f3fee07aa3f441f"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "9259830970158e341ed80d4a76cea978"
server_key_iv: "6085ea32b4a8aaf4b31a8b3547eb6078"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.439116 1689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:41.454406 1689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:41.456843 1689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:41.457027 1689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:41.457523 1689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:41.458303 1689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:41.458369 1689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.458428 1689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:41.458472 1689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.466461 1689 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:42205
I20260810 22:56:41.466493 1828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:42205 every 8 connection(s)
I20260810 22:56:41.467214 1689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:41.470959 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 1689
I20260810 22:56:41.471081 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:41.471434 30010 external_mini_cluster.cc:1428] Setting key b873a9235a3fa41e34f227605ce48352
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:41.482017 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.468906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:53655 (local address 127.29.78.190:35179)
0810 22:56:41.469043 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:41.469046 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:41.469835 (+ 789us) server_negotiation.cc:407] Connection header received
0810 22:56:41.471460 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:41.471463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:41.471532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:41.471621 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:41.473076 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.473601 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.475268 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.475662 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.475783 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.478227 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:41.478231 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:41.478234 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:41.478262 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:41.479924 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.480418 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.480423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.480427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.480525 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.480841 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.480846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.480847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.481088 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:41.481187 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:41.481651 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 22:56:41.481793 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:41.482697 1831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.469241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35179 (local address 127.29.78.129:53655)
0810 22:56:41.469682 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:41.469720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:41.471222 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:41.471825 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:41.471834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:41.472311 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:41.472905 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:41.472925 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.473735 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.473742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:41.475149 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:41.475159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.475807 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.475810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:41.475986 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.476556 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:41.476571 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:41.478070 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:41.480060 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:41.480065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:41.480075 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:41.480311 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:41.480613 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:41.480615 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:41.480616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:41.480737 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:41.481193 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:41.481197 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:41.481408 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 22:56:41.481550 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 22:56:41.481784 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":129,"threads_started":1}
I20260810 22:56:41.483960 1829 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35179
I20260810 22:56:41.484220 1829 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:41.484794 1829 heartbeater.cc:511] Master 127.29.78.190:35179 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 22:56:41.486447 1615 ts_manager.cc:194] Registered new tserver with Master: 45f9ddbb798d4a1d9f3fee07aa3f441f (127.29.78.129:42205)
I20260810 22:56:41.488148 1615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:53655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:41.500439 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35179
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:41.602921 1834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:41.603149 1834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:41.603188 1834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:41.606858 1834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:41.606904 1834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:41.606998 1834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:41.611902 1834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:41.612995 1834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:41.614050 1834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:41.623272 1839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:41.623287 1840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:41.623279 1843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:41.623272 1834 server_base.cc:1034] running on GCE node
I20260810 22:56:41.623823 1834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:41.624396 1834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:41.625581 1834 hybrid_clock.cc:648] HybridClock initialized: now 1786402601625563 us; error 32 us; skew 500 ppm
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:41.628556 1834 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:41.629348 1834 webserver.cc:460] Webserver started at http://127.29.78.130:44505/ using document root <none> and password file <none>
I20260810 22:56:41.629874 1834 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:41.629936 1834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:41.630092 1834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:41.631896 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "54b1d72b5e3746b6acf702fee1e22f8f"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "39adbccbdd3a19d293d9307f64c681b3"
server_key_iv: "9d470182f6fdc11de9a1826b1e81d95e"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.632397 1834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "54b1d72b5e3746b6acf702fee1e22f8f"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "39adbccbdd3a19d293d9307f64c681b3"
server_key_iv: "9d470182f6fdc11de9a1826b1e81d95e"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.635990 1834 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:41.638149 1850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.639230 1834 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:56:41.639344 1834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "54b1d72b5e3746b6acf702fee1e22f8f"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "39adbccbdd3a19d293d9307f64c681b3"
server_key_iv: "9d470182f6fdc11de9a1826b1e81d95e"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.639442 1834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:41.653559 1834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:41.656924 1834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:41.657120 1834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:41.657721 1834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:41.658653 1834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:41.658699 1834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.658757 1834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:41.658789 1834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.667091 1834 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:43385
I20260810 22:56:41.667160 1979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:43385 every 8 connection(s)
I20260810 22:56:41.667953 1834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:41.676492 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 1834
I20260810 22:56:41.676642 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:41.676923 30010 external_mini_cluster.cc:1428] Setting key 138796e1f71033f8b9f31a554eecab99
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:41.682418 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.669723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:46023 (local address 127.29.78.190:35179)
0810 22:56:41.669907 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:41.669912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:41.670702 (+ 790us) server_negotiation.cc:407] Connection header received
0810 22:56:41.672108 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:41.672111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:41.672157 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:41.672229 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:41.673478 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.674134 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.675338 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.675559 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.675637 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.678390 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:41.678393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:41.678395 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:41.678422 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:41.680022 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.680683 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.680689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.680691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.680791 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.681186 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.681192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.681194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.681415 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:41.681530 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:41.682209 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 22:56:41.682309 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:56:41.683614 1982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.670099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35179 (local address 127.29.78.130:46023)
0810 22:56:41.670548 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:41.670589 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:41.671919 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:41.672393 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:41.672399 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:41.672759 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:41.673287 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:41.673323 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.674251 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.674255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:41.675235 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:41.675245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.675699 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.675702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:41.675849 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.676483 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:41.676498 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:41.678194 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:41.680162 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:41.680166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:41.680178 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:41.680550 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:41.680915 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:41.680919 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:41.680921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:41.681079 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:41.681528 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:41.681544 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:41.681947 (+ 403us) client_negotiation.cc:769] Sending connection context
0810 22:56:41.682168 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 22:56:41.682516 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":133,"threads_started":1}
I20260810 22:56:41.684851 1980 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35179
I20260810 22:56:41.685101 1980 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:41.685591 1980 heartbeater.cc:511] Master 127.29.78.190:35179 requested a full tablet report, sending...
I20260810 22:56:41.686687 1615 ts_manager.cc:194] Registered new tserver with Master: 54b1d72b5e3746b6acf702fee1e22f8f (127.29.78.130:43385)
I20260810 22:56:41.687239 1615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:46023
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:41.705181 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35179
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:41.811815 1985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:41.812093 1985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:41.812177 1985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:41.816187 1985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:41.816288 1985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:41.816376 1985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:41.821444 1985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:41.822422 1985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:41.823350 1985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:41.833194 1993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:41.833252 1995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:41.833214 1992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:41.833209 1985 server_base.cc:1034] running on GCE node
I20260810 22:56:41.833770 1985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:41.834386 1985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:41.835546 1985 hybrid_clock.cc:648] HybridClock initialized: now 1786402601835517 us; error 37 us; skew 500 ppm
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:41.838238 1985 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:41.839043 1985 webserver.cc:460] Webserver started at http://127.29.78.131:34133/ using document root <none> and password file <none>
I20260810 22:56:41.839604 1985 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:41.839665 1985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:41.839886 1985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:41.841794 1985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "8ad9a2fd7e9e42a6a2cd2a7627df0924"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "e0907da95a2574378377d9c71d767e8b"
server_key_iv: "185fde35667945f53f5fc0114d3f98bc"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.842481 1985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "8ad9a2fd7e9e42a6a2cd2a7627df0924"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "e0907da95a2574378377d9c71d767e8b"
server_key_iv: "185fde35667945f53f5fc0114d3f98bc"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.846259 1985 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 22:56:41.848613 2002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.849752 1985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:41.849902 1985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "8ad9a2fd7e9e42a6a2cd2a7627df0924"
format_stamp: "Formatted at 2026-08-10 22:56:41 on dist-test-slave-6nmv"
server_key: "e0907da95a2574378377d9c71d767e8b"
server_key_iv: "185fde35667945f53f5fc0114d3f98bc"
server_key_version: "encryptionkey@0"
I20260810 22:56:41.850008 1985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:41.863133 1985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:41.866298 1985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:41.866475 1985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:41.866989 1985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:41.867750 1985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:41.867813 1985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.867874 1985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:41.867918 1985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:41.877452 1985 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41487
I20260810 22:56:41.877475 2145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41487 every 8 connection(s)
I20260810 22:56:41.878346 1985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:41.881177 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 1985
I20260810 22:56:41.881302 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:41.881548 30010 external_mini_cluster.cc:1428] Setting key caba5783700f5e1da95df3ed375c54a1
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:41.890635 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.880079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:34029 (local address 127.29.78.190:35179)
0810 22:56:41.880205 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:41.880208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:41.880976 (+ 768us) server_negotiation.cc:407] Connection header received
0810 22:56:41.882510 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:41.882514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:41.882565 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:41.882647 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:41.883784 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.884243 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.885273 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.885475 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.885562 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.887850 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:41.887852 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:41.887854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:41.887878 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:41.888961 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.889426 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.889428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.889429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.889495 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.889769 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.889771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.889772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.889979 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:41.890055 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:41.890442 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 22:56:41.890533 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:41.891417 2148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.880427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35179 (local address 127.29.78.131:34029)
0810 22:56:41.880833 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:41.880870 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:41.882288 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:41.882836 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:41.882843 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:41.883166 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:41.883669 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:41.883683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.884336 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.884338 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:41.885171 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:41.885180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.885608 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.885611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:41.885742 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.886339 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:41.886355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:41.887715 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:41.889072 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:41.889076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:41.889086 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:41.889347 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:41.889577 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:41.889579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:41.889580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:41.889700 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:41.890066 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:41.890070 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:41.890270 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 22:56:41.890413 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 22:56:41.890619 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":106,"threads_started":1}
I20260810 22:56:41.892457 2146 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35179
I20260810 22:56:41.892642 2146 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:41.893055 2146 heartbeater.cc:511] Master 127.29.78.190:35179 requested a full tablet report, sending...
I20260810 22:56:41.894003 1615 ts_manager.cc:194] Registered new tserver with Master: 8ad9a2fd7e9e42a6a2cd2a7627df0924 (127.29.78.131:41487)
I20260810 22:56:41.894737 1615 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:34029
I20260810 22:56:41.895198 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:41.920428 2154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.911398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.29.78.129:42205)
0810 22:56:41.911817 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:41.911827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:41.911850 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:56:41.911896 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:41.911900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:41.912079 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:41.912236 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:41.912959 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.913682 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.914515 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.914889 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.915013 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.917268 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:41.917277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:41.917288 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:41.917309 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:41.918707 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.919141 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.919146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.919147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.919222 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.919496 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.919507 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.919509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.919742 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:41.919953 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:41.920136 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 22:56:41.920275 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":143,"threads_started":1}
Aug 10 22:56:41 dist-test-slave-6nmv krb5kdc[1566](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:41.930797 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:41.922961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.29.78.190:35179)
0810 22:56:41.923093 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:41.923099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:41.923115 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:56:41.923239 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:41.923242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:41.923301 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:41.923384 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:41.923922 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.924386 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.925130 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:41.925364 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:41.925470 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:41.927878 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:41.927881 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:41.927884 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:41.927914 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:41.929091 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.929469 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.929474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.929475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.929581 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:41.929900 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:41.929903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:41.929905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:41.930124 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:41.930268 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:41.930491 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 22:56:41.930617 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 22:56:41.931236 1615 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48216
I20260810 22:56:41.931900 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 1689
I20260810 22:56:41.940750 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 1834
I20260810 22:56:41.947070 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 1985
I20260810 22:56:41.953779 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 1580
2026-08-10T22:56:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2095 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:42 dist-test-slave-6nmv krb5kdc[2159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:42 dist-test-slave-6nmv krb5kdc[2159](info): set up 2 sockets
Aug 10 22:56:42 dist-test-slave-6nmv krb5kdc[2159](info): commencing operation
krb5kdc: starting...
W20260810 22:56:43.000978 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:43.122467 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33497
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:33497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:43.228327 2178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:43.228595 2178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:43.228663 2178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:43.232476 2178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:43.232559 2178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:43.232599 2178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:43.232635 2178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:43.232664 2178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:43.237334 2178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:33497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:43.238245 2178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:43.239113 2178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:43.246832 2185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:43.246865 2188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:43.246937 2178 server_base.cc:1034] running on GCE node
W20260810 22:56:43.246869 2186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:43.247685 2178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:43.248687 2178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:43.249886 2178 hybrid_clock.cc:648] HybridClock initialized: now 1786402603249861 us; error 29 us; skew 500 ppm
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:43.252568 2178 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:43.253226 2178 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:43.253314 2178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:43.253532 2178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:43.255525 2178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "caaaba2baab64604b82a482788093dc3"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "f2249a1bbceddfb8c532b0af122e50ff"
server_key_iv: "4e397d062ad9a256c6bbfc31031cc5cb"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.256093 2178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "caaaba2baab64604b82a482788093dc3"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "f2249a1bbceddfb8c532b0af122e50ff"
server_key_iv: "4e397d062ad9a256c6bbfc31031cc5cb"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.259462 2178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:43.261778 2195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.262961 2178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:43.263096 2178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "caaaba2baab64604b82a482788093dc3"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "f2249a1bbceddfb8c532b0af122e50ff"
server_key_iv: "4e397d062ad9a256c6bbfc31031cc5cb"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.263197 2178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:43.272362 2178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:43.274982 2178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:43.275173 2178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:43.282265 2178 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:33497
I20260810 22:56:43.282580 2254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:33497 every 8 connection(s)
I20260810 22:56:43.282931 2178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:43.285494 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:43.288147 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 2178
I20260810 22:56:43.288287 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:43.288571 30010 external_mini_cluster.cc:1428] Setting key d80eb03196c7f592ef189a8538047ad5
I20260810 22:56:43.291260 2255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3: Bootstrap starting.
I20260810 22:56:43.293883 2255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:43.294690 2255 log.cc:824] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:43.296533 2255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3: No bootstrap required, opened a new log
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:43.300415 2255 raft_consensus.cc:348] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaba2baab64604b82a482788093dc3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33497 } }
I20260810 22:56:43.300681 2255 raft_consensus.cc:374] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:43.300735 2255 raft_consensus.cc:729] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caaaba2baab64604b82a482788093dc3, State: Initialized, Role: FOLLOWER
I20260810 22:56:43.301141 2255 consensus_queue.cc:260] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaba2baab64604b82a482788093dc3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33497 } }
I20260810 22:56:43.301285 2255 raft_consensus.cc:388] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:43.301373 2255 raft_consensus.cc:482] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:43.301478 2255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:43.302377 2255 raft_consensus.cc:504] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaba2baab64604b82a482788093dc3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33497 } }
I20260810 22:56:43.302662 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.289913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.29.78.190:33497)
0810 22:56:43.290310 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.290324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.290355 (+ 31us) server_negotiation.cc:407] Connection header received
0810 22:56:43.290910 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.290942 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.291253 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.291583 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:43.292466 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.293356 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.294474 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.294954 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.295108 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.297547 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:43.297552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:43.297567 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:43.297596 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:43.299953 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.300338 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.300343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.300347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.300479 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.300764 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.300768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.300772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.301126 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:43.301310 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:43.301620 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 22:56:43.301948 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":120,"threads_started":1}
I20260810 22:56:43.302695 2255 leader_election.cc:302] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caaaba2baab64604b82a482788093dc3; no voters:
I20260810 22:56:43.302976 2255 leader_election.cc:288] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:43.303050 2260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:43.303229 2260 raft_consensus.cc:686] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [term 1 LEADER]: Becoming Leader. State: Replica: caaaba2baab64604b82a482788093dc3, State: Running, Role: LEADER
I20260810 22:56:43.303591 2260 consensus_queue.cc:237] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaba2baab64604b82a482788093dc3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33497 } }
I20260810 22:56:43.304131 2255 sys_catalog.cc:564] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:43.304662 2260 sys_catalog.cc:455] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caaaba2baab64604b82a482788093dc3. Latest consensus state: current_term: 1 leader_uuid: "caaaba2baab64604b82a482788093dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaba2baab64604b82a482788093dc3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33497 } } }
I20260810 22:56:43.304770 2260 sys_catalog.cc:458] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:43.305238 2266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:43.305747 2261 sys_catalog.cc:455] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caaaba2baab64604b82a482788093dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caaaba2baab64604b82a482788093dc3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33497 } } }
I20260810 22:56:43.305965 2261 sys_catalog.cc:458] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:43.307960 2266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:43.313215 2266 catalog_manager.cc:1347] Generated new cluster ID: 80bfaf057333440a8337852285716fab
I20260810 22:56:43.313287 2266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:43.329263 2266 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:43.330233 2266 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:43.339340 2266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3: Generated new TSK 0
I20260810 22:56:43.340044 2266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:43.374910 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33497
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:43.478440 2284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:43.478657 2284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:43.478696 2284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:43.482470 2284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:43.482513 2284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:43.482602 2284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:43.487344 2284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:43.488229 2284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:43.489099 2284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:43.497879 2291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:43.497887 2294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:43.497884 2292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:43.498111 2284 server_base.cc:1034] running on GCE node
I20260810 22:56:43.498538 2284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:43.499039 2284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:43.500203 2284 hybrid_clock.cc:648] HybridClock initialized: now 1786402603500180 us; error 34 us; skew 500 ppm
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:43.502843 2284 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:43.503479 2284 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:43.503543 2284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:43.503760 2284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:43.505896 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "74285d9cd4d6408bbb8b7a118f190609"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "117acc506a1271adc0a96978e0394368"
server_key_iv: "90f86891c8e84571b8f29dd7f7fc4db8"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.506455 2284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "74285d9cd4d6408bbb8b7a118f190609"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "117acc506a1271adc0a96978e0394368"
server_key_iv: "90f86891c8e84571b8f29dd7f7fc4db8"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.509599 2284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 22:56:43.511868 2301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.512979 2284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:43.513087 2284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "74285d9cd4d6408bbb8b7a118f190609"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "117acc506a1271adc0a96978e0394368"
server_key_iv: "90f86891c8e84571b8f29dd7f7fc4db8"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.513193 2284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:43.524428 2284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:43.527213 2284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:43.527405 2284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:43.527907 2284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:43.528641 2284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:43.528683 2284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.528738 2284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:43.528767 2284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.537108 2284 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38021
I20260810 22:56:43.537132 2421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38021 every 8 connection(s)
I20260810 22:56:43.537590 2284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:43.541145 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 2284
I20260810 22:56:43.541260 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:43.541531 30010 external_mini_cluster.cc:1428] Setting key 3b50e67a40385b87ea834352ca136942
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:43.552232 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.539134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:41285 (local address 127.29.78.190:33497)
0810 22:56:43.539268 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.539273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.540061 (+ 788us) server_negotiation.cc:407] Connection header received
0810 22:56:43.541247 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.541250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.541305 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.541389 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:43.542691 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.543231 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.544872 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.545338 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.545490 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.548382 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:43.548384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:43.548386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:43.548408 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:43.549991 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.550634 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.550637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.550638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.550705 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.551122 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.551123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.551125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.551273 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:43.551341 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:43.552019 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 22:56:43.552115 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:43.553406 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.539512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33497 (local address 127.29.78.129:41285)
0810 22:56:43.539918 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:43.539955 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:43.541059 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:43.541571 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:43.541578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:43.542005 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:43.542533 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.542546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.543359 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.543365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:43.544697 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.544711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.545509 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.545512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:43.545643 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.546261 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:43.546275 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:43.548222 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:43.550122 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:43.550129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:43.550143 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:43.550538 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:43.550797 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:43.550800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:43.550803 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:43.551011 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:43.551380 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:43.551387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:43.551727 (+ 340us) client_negotiation.cc:769] Sending connection context
0810 22:56:43.551992 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 22:56:43.552360 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":105,"threads_started":1}
I20260810 22:56:43.554616 2422 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33497
I20260810 22:56:43.554862 2422 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:43.555339 2422 heartbeater.cc:511] Master 127.29.78.190:33497 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 22:56:43.556849 2217 ts_manager.cc:194] Registered new tserver with Master: 74285d9cd4d6408bbb8b7a118f190609 (127.29.78.129:38021)
I20260810 22:56:43.558048 2217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:41285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:43.570744 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33497
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:43.672542 2427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:43.672791 2427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:43.672847 2427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:43.676564 2427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:43.676612 2427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:43.676699 2427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:43.681154 2427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:43.682094 2427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:43.682966 2427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:43.691437 2436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:43.691437 2434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:43.691485 2433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:43.691687 2427 server_base.cc:1034] running on GCE node
I20260810 22:56:43.692157 2427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:43.692767 2427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:43.693892 2427 hybrid_clock.cc:648] HybridClock initialized: now 1786402603693869 us; error 31 us; skew 500 ppm
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:43.696619 2427 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:43.697196 2427 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:43.697247 2427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:43.697389 2427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:43.699164 2427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "5ace8441e5674caa8a7232723ea284da"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "38be2772761315c2fa1a85c53312c82c"
server_key_iv: "49a97fd9055f228fb8fcc39d0773b61c"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.699615 2427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "5ace8441e5674caa8a7232723ea284da"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "38be2772761315c2fa1a85c53312c82c"
server_key_iv: "49a97fd9055f228fb8fcc39d0773b61c"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.702674 2427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:56:43.704778 2443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.706173 2427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:43.706314 2427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "5ace8441e5674caa8a7232723ea284da"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "38be2772761315c2fa1a85c53312c82c"
server_key_iv: "49a97fd9055f228fb8fcc39d0773b61c"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.706399 2427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:43.727346 2427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:43.730283 2427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:43.730422 2427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:43.730942 2427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:43.731774 2427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:43.731820 2427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.731887 2427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:43.731923 2427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.740670 2427 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:35905
I20260810 22:56:43.740689 2562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:35905 every 8 connection(s)
I20260810 22:56:43.741207 2427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:43.746481 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 2427
I20260810 22:56:43.746629 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:43.746913 30010 external_mini_cluster.cc:1428] Setting key 12940d585c393fe8d030afef1938e206
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:43.755653 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.742817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34523 (local address 127.29.78.190:33497)
0810 22:56:43.742957 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.742961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.743768 (+ 807us) server_negotiation.cc:407] Connection header received
0810 22:56:43.745094 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.745098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.745159 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.745267 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:43.746532 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.747155 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.748182 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.748437 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.748512 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.751762 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:43.751768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:43.751774 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:43.751808 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:43.753419 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.753988 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.753994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.754005 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.754098 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.754421 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.754427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.754431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.754688 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:43.754800 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:43.755286 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 22:56:43.755431 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:56:43.756317 2565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.743194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33497 (local address 127.29.78.130:34523)
0810 22:56:43.743625 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:43.743665 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:43.744868 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:43.745439 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:43.745445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:43.745800 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:43.746372 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.746394 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.747286 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.747288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:43.748084 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.748092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.748591 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.748597 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:43.748827 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.749509 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:43.749535 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:43.751561 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:43.753549 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:43.753555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:43.753570 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:43.753895 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:43.754191 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:43.754194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:43.754195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:43.754316 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:43.754792 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:43.754796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:43.755029 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 22:56:43.755172 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 22:56:43.755413 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260810 22:56:43.757375 2563 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33497
I20260810 22:56:43.757654 2563 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:56:43.758211 2563 heartbeater.cc:511] Master 127.29.78.190:33497 requested a full tablet report, sending...
I20260810 22:56:43.759207 2217 ts_manager.cc:194] Registered new tserver with Master: 5ace8441e5674caa8a7232723ea284da (127.29.78.130:35905)
I20260810 22:56:43.759723 2217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:34523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:43.774466 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33497
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:43.878393 2568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:43.878798 2568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:43.878868 2568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:43.883316 2568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:43.883373 2568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:43.883431 2568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:43.888574 2568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:33497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:43.889537 2568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:43.890545 2568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:43.900115 2575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:43.900163 2574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:43.900221 2577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:43.900238 2568 server_base.cc:1034] running on GCE node
I20260810 22:56:43.900684 2568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:43.901338 2568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:43.902521 2568 hybrid_clock.cc:648] HybridClock initialized: now 1786402603902488 us; error 45 us; skew 500 ppm
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:43.905802 2568 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:43.906651 2568 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:43.906739 2568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:43.906944 2568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:43.909005 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "5f3116a612d0eda8b1199f54a0c61553"
server_key_iv: "8296ebff4ef4f2c0d56aae4549bf44fe"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.909588 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "5f3116a612d0eda8b1199f54a0c61553"
server_key_iv: "8296ebff4ef4f2c0d56aae4549bf44fe"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.913209 2568 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260810 22:56:43.915611 2583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.916939 2568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:43.917062 2568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4"
format_stamp: "Formatted at 2026-08-10 22:56:43 on dist-test-slave-6nmv"
server_key: "5f3116a612d0eda8b1199f54a0c61553"
server_key_iv: "8296ebff4ef4f2c0d56aae4549bf44fe"
server_key_version: "encryptionkey@0"
I20260810 22:56:43.917179 2568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:43.940735 2568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:43.943638 2568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:43.943866 2568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:43.944423 2568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:43.945305 2568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:43.945379 2568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.945449 2568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:43.945498 2568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:43.954320 2568 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:38119
I20260810 22:56:43.954339 2705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:38119 every 8 connection(s)
I20260810 22:56:43.954988 2568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:43.960597 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 2568
I20260810 22:56:43.960690 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:43.960958 30010 external_mini_cluster.cc:1428] Setting key 751b3c8c38fac7829b33b57e8aec3f79
Aug 10 22:56:43 dist-test-slave-6nmv krb5kdc[2159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402603, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:43.968302 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.956797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:33755 (local address 127.29.78.190:33497)
0810 22:56:43.956943 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.956948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.957858 (+ 910us) server_negotiation.cc:407] Connection header received
0810 22:56:43.959182 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.959184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.959228 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.959325 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:43.960486 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.961115 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.962521 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.962717 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.962787 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.965191 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:43.965194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:43.965196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:43.965220 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:43.966560 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.967069 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.967072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.967073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.967145 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.967421 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.967423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.967424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.967587 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:43.967656 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:43.968089 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 22:56:43.968191 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:56:43.969077 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.957221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33497 (local address 127.29.78.131:33755)
0810 22:56:43.957679 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:43.957729 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:43.959007 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:43.959486 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:43.959492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:43.959837 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:43.960307 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.960319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.961239 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.961245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:43.962404 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.962415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.962853 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.962857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:43.963045 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.963615 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:43.963629 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:43.965041 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:43.966697 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:43.966701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:43.966711 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:43.966991 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:43.967218 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:43.967220 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:43.967221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:43.967340 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:43.967666 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:43.967670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:43.967878 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 22:56:43.968047 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 22:56:43.968300 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":109,"threads_started":1}
I20260810 22:56:43.970167 2706 heartbeater.cc:347] Connected to a master server at 127.29.78.190:33497
I20260810 22:56:43.970399 2706 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:43.970821 2706 heartbeater.cc:511] Master 127.29.78.190:33497 requested a full tablet report, sending...
I20260810 22:56:43.971825 2217 ts_manager.cc:194] Registered new tserver with Master: 24d11bcb60ed4466bdfcc0c5e4ca7fd4 (127.29.78.131:38119)
I20260810 22:56:43.972388 2217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:33755
I20260810 22:56:43.974690 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:56:43.983211 2258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.976554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.29.78.190:33497)
0810 22:56:43.976694 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.976698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.976882 (+ 184us) server_negotiation.cc:407] Connection header received
0810 22:56:43.976982 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.976984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.977027 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.977102 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:43.977734 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.978258 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.979007 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:43.979206 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:43.979313 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:43.980455 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:43.980458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:43.980460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:43.980506 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:43.981580 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.982044 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.982049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.982050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.982129 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:43.982380 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:43.982385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:43.982388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:43.982588 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:43.982720 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:43.982936 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 22:56:43.983042 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:56:43.985667 2217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:43.988569 2217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:44.003336 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.996894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:35905 (local address 127.0.0.1:37216)
0810 22:56:43.997729 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:43.997781 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:43.997949 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:43.998512 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:43.998517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:43.998540 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:43.998702 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.998709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.000262 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.000265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.001677 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:44.001686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.002779 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.002783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.002906 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.002966 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 22:56:44.003060 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 22:56:44.003175 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":113,"spinlock_wait_cycles":3072,"thread_start_us":108,"threads_started":1}
I20260810 22:56:44.003610 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.996944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:38119 (local address 127.0.0.1:41398)
0810 22:56:43.997729 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:43.997781 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:43.997946 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:43.998680 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:43.998682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:43.998696 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:43.998804 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.998810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.000301 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.000304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.001884 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:44.001895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.003197 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.003200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.003282 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.003302 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:56:44.003379 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 22:56:44.003453 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":90,"spinlock_wait_cycles":6144,"thread_start_us":123,"threads_started":1}
I20260810 22:56:44.003942 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.997339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.29.78.131:38119)
0810 22:56:43.997797 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.997808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.998091 (+ 283us) server_negotiation.cc:407] Connection header received
0810 22:56:43.998173 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.998179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.998373 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.998535 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:43.998924 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.000114 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.002009 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.002901 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.002997 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.003140 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.003432 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.003795 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":143,"threads_started":1}
I20260810 22:56:44.004370 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.997031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.29.78.130:35905)
0810 22:56:43.998020 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.998033 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.998057 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:56:43.998136 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.998142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.998344 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.998520 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:43.998842 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.000114 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.001800 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.002635 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.003630 (+ 995us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.003892 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.003934 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.004033 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":113,"threads_started":1}
I20260810 22:56:44.004845 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.997652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38021 (local address 127.0.0.1:35448)
0810 22:56:43.998160 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:43.998172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:43.998244 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:43.999244 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:43.999247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:43.999265 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:43.999384 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:43.999394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.000386 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.000389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.001884 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:44.001895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.004419 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.004423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.004517 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.004538 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:56:44.004607 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 22:56:44.004676 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":257,"thread_start_us":81,"threads_started":1}
I20260810 22:56:44.005327 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:43.998404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.29.78.129:38021)
0810 22:56:43.998843 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:43.998854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:43.998875 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:56:43.998953 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:43.998957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:43.999086 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:43.999261 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:43.999518 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.000258 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.002037 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.004069 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.004247 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.004368 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.005033 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.005127 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":128,"threads_started":1}
I20260810 22:56:44.006973 2493 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7359ec297b41f18a36a10a1cd06177 (DEFAULT_TABLE table=test-table [id=79411cf23ea045c5b897eb09b714062d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:44.006927 2633 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7359ec297b41f18a36a10a1cd06177 (DEFAULT_TABLE table=test-table [id=79411cf23ea045c5b897eb09b714062d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:44.007906 2493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7359ec297b41f18a36a10a1cd06177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:44.007947 2633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7359ec297b41f18a36a10a1cd06177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:44.008200 2351 tablet_service.cc:1467] Processing CreateTablet for tablet 1e7359ec297b41f18a36a10a1cd06177 (DEFAULT_TABLE table=test-table [id=79411cf23ea045c5b897eb09b714062d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:44.009094 2351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e7359ec297b41f18a36a10a1cd06177. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:44.014034 2725 tablet_bootstrap.cc:492] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Bootstrap starting.
I20260810 22:56:44.015374 2726 tablet_bootstrap.cc:492] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Bootstrap starting.
I20260810 22:56:44.015736 2727 tablet_bootstrap.cc:492] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: Bootstrap starting.
I20260810 22:56:44.016815 2725 tablet_bootstrap.cc:654] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:44.017611 2726 tablet_bootstrap.cc:654] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:44.017663 2727 tablet_bootstrap.cc:654] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:44.018440 2725 log.cc:824] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:44.018851 2727 log.cc:824] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:44.018918 2726 log.cc:824] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:44.020694 2726 tablet_bootstrap.cc:492] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: No bootstrap required, opened a new log
I20260810 22:56:44.020737 2725 tablet_bootstrap.cc:492] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: No bootstrap required, opened a new log
I20260810 22:56:44.020737 2727 tablet_bootstrap.cc:492] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: No bootstrap required, opened a new log
I20260810 22:56:44.020893 2726 ts_tablet_manager.cc:1397] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:56:44.020896 2725 ts_tablet_manager.cc:1397] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:56:44.020896 2727 ts_tablet_manager.cc:1397] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:56:44.023693 2726 raft_consensus.cc:348] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.023928 2726 raft_consensus.cc:374] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:44.023980 2726 raft_consensus.cc:729] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24d11bcb60ed4466bdfcc0c5e4ca7fd4, State: Initialized, Role: FOLLOWER
I20260810 22:56:44.024394 2726 consensus_queue.cc:260] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.024562 2727 raft_consensus.cc:348] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.024562 2725 raft_consensus.cc:348] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.024809 2727 raft_consensus.cc:374] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:44.024809 2725 raft_consensus.cc:374] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:44.024869 2727 raft_consensus.cc:729] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74285d9cd4d6408bbb8b7a118f190609, State: Initialized, Role: FOLLOWER
I20260810 22:56:44.024869 2725 raft_consensus.cc:729] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ace8441e5674caa8a7232723ea284da, State: Initialized, Role: FOLLOWER
I20260810 22:56:44.025238 2706 heartbeater.cc:503] Master 127.29.78.190:33497 was elected leader, sending a full tablet report...
I20260810 22:56:44.025533 2726 ts_tablet_manager.cc:1428] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:56:44.025334 2725 consensus_queue.cc:260] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.025992 2727 consensus_queue.cc:260] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.026494 2725 ts_tablet_manager.cc:1428] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:56:44.026822 2563 heartbeater.cc:503] Master 127.29.78.190:33497 was elected leader, sending a full tablet report...
I20260810 22:56:44.026899 2727 ts_tablet_manager.cc:1428] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:56:44.027244 2422 heartbeater.cc:503] Master 127.29.78.190:33497 was elected leader, sending a full tablet report...
W20260810 22:56:44.038643 2423 tablet.cc:2369] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:56:44.205969 2707 tablet.cc:2369] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:56:44.242273 2564 tablet.cc:2369] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:44.321560 2734 raft_consensus.cc:482] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:44.321780 2734 raft_consensus.cc:504] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.323036 2734 leader_election.cc:288] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ace8441e5674caa8a7232723ea284da (127.29.78.130:35905), 74285d9cd4d6408bbb8b7a118f190609 (127.29.78.129:38021)
I20260810 22:56:44.326735 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.323423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:37739 (local address 127.29.78.130:35905)
0810 22:56:44.323560 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:44.323566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:44.323581 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:56:44.323635 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:44.323637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:44.323683 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:44.323765 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:44.324103 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.324631 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.325705 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.326184 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.326269 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.326341 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.326554 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.326620 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:44.326766 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.323408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:58277 (local address 127.29.78.129:38021)
0810 22:56:44.323532 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:44.323536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:44.323796 (+ 260us) server_negotiation.cc:407] Connection header received
0810 22:56:44.323865 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:44.323867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:44.323912 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:44.323996 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:44.324269 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.324737 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.325763 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.326216 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.326338 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.326386 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.326600 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.326671 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:56:44.326735 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.323232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:35905 (local address 127.29.78.131:37739)
0810 22:56:44.323349 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:44.323366 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:44.323458 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:44.323817 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:44.323821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:44.323845 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:44.323982 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:44.323990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.324738 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.324741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.325578 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:44.325587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.326293 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.326299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.326413 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.326452 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 22:56:44.326506 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 22:56:44.326583 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":2}
I20260810 22:56:44.326751 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.323321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38021 (local address 127.29.78.131:58277)
0810 22:56:44.323699 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:44.323712 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:44.323793 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:44.324017 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:44.324018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:44.324030 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:44.324166 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:44.324172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.324843 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.324845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.325614 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:44.325622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.326351 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.326357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:44.326460 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.326477 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 22:56:44.326565 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:56:44.326622 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":14,"thread_start_us":113,"threads_started":1}
I20260810 22:56:44.327564 2513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7359ec297b41f18a36a10a1cd06177" candidate_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ace8441e5674caa8a7232723ea284da" is_pre_election: true
I20260810 22:56:44.327564 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7359ec297b41f18a36a10a1cd06177" candidate_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74285d9cd4d6408bbb8b7a118f190609" is_pre_election: true
I20260810 22:56:44.327869 2513 raft_consensus.cc:2454] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24d11bcb60ed4466bdfcc0c5e4ca7fd4 in term 0.
I20260810 22:56:44.327858 2372 raft_consensus.cc:2454] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24d11bcb60ed4466bdfcc0c5e4ca7fd4 in term 0.
I20260810 22:56:44.328349 2588 leader_election.cc:302] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24d11bcb60ed4466bdfcc0c5e4ca7fd4, 5ace8441e5674caa8a7232723ea284da; no voters:
I20260810 22:56:44.328622 2734 raft_consensus.cc:2781] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:44.328712 2734 raft_consensus.cc:482] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:44.328766 2734 raft_consensus.cc:3037] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:44.329720 2734 raft_consensus.cc:504] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.330063 2734 leader_election.cc:288] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [CANDIDATE]: Term 1 election: Requested vote from peers 5ace8441e5674caa8a7232723ea284da (127.29.78.130:35905), 74285d9cd4d6408bbb8b7a118f190609 (127.29.78.129:38021)
I20260810 22:56:44.330431 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7359ec297b41f18a36a10a1cd06177" candidate_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74285d9cd4d6408bbb8b7a118f190609"
I20260810 22:56:44.330484 2513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e7359ec297b41f18a36a10a1cd06177" candidate_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ace8441e5674caa8a7232723ea284da"
I20260810 22:56:44.330561 2372 raft_consensus.cc:3037] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:44.330614 2513 raft_consensus.cc:3037] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:44.331568 2372 raft_consensus.cc:2454] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24d11bcb60ed4466bdfcc0c5e4ca7fd4 in term 1.
I20260810 22:56:44.331652 2513 raft_consensus.cc:2454] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24d11bcb60ed4466bdfcc0c5e4ca7fd4 in term 1.
I20260810 22:56:44.331923 2584 leader_election.cc:302] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24d11bcb60ed4466bdfcc0c5e4ca7fd4, 74285d9cd4d6408bbb8b7a118f190609; no voters:
I20260810 22:56:44.332115 2734 raft_consensus.cc:2781] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:44.332346 2734 raft_consensus.cc:686] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 24d11bcb60ed4466bdfcc0c5e4ca7fd4, State: Running, Role: LEADER
I20260810 22:56:44.332757 2734 consensus_queue.cc:237] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } }
I20260810 22:56:44.335778 2217 catalog_manager.cc:5495] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24d11bcb60ed4466bdfcc0c5e4ca7fd4 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24d11bcb60ed4466bdfcc0c5e4ca7fd4" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:44.363411 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.360369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.29.78.131:38119)
0810 22:56:44.360473 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:44.360477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:44.360511 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:56:44.360640 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:44.360642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:44.360687 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:44.360765 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:44.360993 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.361430 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.362214 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:44.362391 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:44.362459 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:44.362702 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:44.363075 (+ 373us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:44.363156 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.363260 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.363303 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 22:56:44.368945 2513 raft_consensus.cc:1261] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 1 FOLLOWER]: Refusing update from remote peer 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:44.368945 2372 raft_consensus.cc:1261] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609 [term 1 FOLLOWER]: Refusing update from remote peer 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:44.369659 2741 consensus_queue.cc:1035] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74285d9cd4d6408bbb8b7a118f190609" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:44.369844 2734 consensus_queue.cc:1035] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ace8441e5674caa8a7232723ea284da" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:44.377938 2747 mvcc.cc:205] Tried to move back new op lower bound from 7317105067488071680 to 7317105067349487616. Current Snapshot: MvccSnapshot[applied={T|T < 7317105067488071680}]
I20260810 22:56:44.385789 2217 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:44.386021 2217 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:44.388357 2217 catalog_manager.cc:5782] T 00000000000000000000000000000000 P caaaba2baab64604b82a482788093dc3: Sending DeleteTablet for 3 replicas of tablet 1e7359ec297b41f18a36a10a1cd06177
I20260810 22:56:44.389079 2493 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7359ec297b41f18a36a10a1cd06177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:44 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:37216
I20260810 22:56:44.389156 2351 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7359ec297b41f18a36a10a1cd06177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:44 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35448
I20260810 22:56:44.389566 2633 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e7359ec297b41f18a36a10a1cd06177 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:44 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:41398
I20260810 22:56:44.389997 2755 tablet_replica.cc:330] T 1e7359ec297b41f18a36a10a1cd06177 P 74285d9cd4d6408bbb8b7a118f190609: stopping tablet replica
I20260810 22:56:44.389995 2756 tablet_replica.cc:330] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: stopping tablet replica
I20260810 22:56:44.390132 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 2284
I20260810 22:56:44.390455 2756 raft_consensus.cc:2229] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:44.390789 2756 raft_consensus.cc:2258] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:44.391165 2758 tablet_replica.cc:330] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: stopping tablet replica
I20260810 22:56:44.391433 2758 raft_consensus.cc:2229] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:44.391800 2758 raft_consensus.cc:2258] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:44.393556 2756 ts_tablet_manager.cc:1905] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:44.396378 2756 ts_tablet_manager.cc:1918] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:44.396472 2756 log.cc:1196] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/1e7359ec297b41f18a36a10a1cd06177
I20260810 22:56:44.396767 2756 ts_tablet_manager.cc:1939] T 1e7359ec297b41f18a36a10a1cd06177 P 5ace8441e5674caa8a7232723ea284da: Deleting consensus metadata
I20260810 22:56:44.397970 2200 catalog_manager.cc:4841] TS 5ace8441e5674caa8a7232723ea284da (127.29.78.130:35905): tablet 1e7359ec297b41f18a36a10a1cd06177 (table test-table [id=79411cf23ea045c5b897eb09b714062d]) successfully deleted
I20260810 22:56:44.398602 2758 ts_tablet_manager.cc:1905] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:56:44.400278 2196 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:38021 (error 108)
I20260810 22:56:44.400990 2758 ts_tablet_manager.cc:1918] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:44.401096 2758 log.cc:1196] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/1e7359ec297b41f18a36a10a1cd06177
I20260810 22:56:44.401383 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.401022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38021 (local address 127.0.0.1:35464)
0810 22:56:44.401147 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:44.401247 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38021: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 22:56:44.401417 2758 ts_tablet_manager.cc:1939] T 1e7359ec297b41f18a36a10a1cd06177 P 24d11bcb60ed4466bdfcc0c5e4ca7fd4: Deleting consensus metadata
I20260810 22:56:44.401499 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 2427
W20260810 22:56:44.404613 2196 catalog_manager.cc:4584] TS 74285d9cd4d6408bbb8b7a118f190609 (127.29.78.129:38021): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e7359ec297b41f18a36a10a1cd06177: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38021: connect: Connection refused (error 111)
I20260810 22:56:44.404816 2196 catalog_manager.cc:4841] TS 24d11bcb60ed4466bdfcc0c5e4ca7fd4 (127.29.78.131:38119): tablet 1e7359ec297b41f18a36a10a1cd06177 (table test-table [id=79411cf23ea045c5b897eb09b714062d]) successfully deleted
I20260810 22:56:44.409942 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 2568
I20260810 22:56:44.416257 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 2178
2026-08-10T22:56:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2463 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T22:56:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:44Z Disabled control of system clock
I20260810 22:56:44.447482 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45695
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:56:44.552201 2768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:44.552434 2768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:44.552469 2768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:44.556273 2768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:44.556321 2768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:44.556335 2768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:44.556350 2768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:44.556363 2768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:44.560393 2768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45695
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:44.561249 2768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:44.562610 2768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:44.570290 2773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:44.570312 2776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:44.570307 2774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:44.570645 2768 server_base.cc:1034] running on GCE node
I20260810 22:56:44.571175 2768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:44.572216 2768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:44.573406 2768 hybrid_clock.cc:648] HybridClock initialized: now 1786402604573387 us; error 31 us; skew 500 ppm
I20260810 22:56:44.582469 2768 webserver.cc:460] Webserver started at http://127.29.78.190:36345/ using document root <none> and password file <none>
I20260810 22:56:44.582995 2768 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:44.583077 2768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:44.583303 2768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:44.585235 2768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "37220803399df56cfff58524ade19d7d"
server_key_iv: "7ad0d0597c93e412007810d292f8f66d"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.585799 2768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "37220803399df56cfff58524ade19d7d"
server_key_iv: "7ad0d0597c93e412007810d292f8f66d"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.589391 2768 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 22:56:44.591702 2783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:44.592921 2768 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:56:44.593050 2768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "37220803399df56cfff58524ade19d7d"
server_key_iv: "7ad0d0597c93e412007810d292f8f66d"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.593149 2768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:44.597318 2768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:44.597901 2768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:44.598063 2768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:44.605748 2768 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45695
I20260810 22:56:44.605798 2839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45695 every 8 connection(s)
I20260810 22:56:44.606802 2768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:44.609925 2840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:44.612386 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 2768
I20260810 22:56:44.612501 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:44.612804 30010 external_mini_cluster.cc:1428] Setting key 1d08222913b7df46d5dfaf0e87cbb757
I20260810 22:56:44.615602 2840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7: Bootstrap starting.
I20260810 22:56:44.617837 2840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:44.618109 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.614211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.29.78.190:45695)
0810 22:56:44.614776 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:44.614788 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:44.614819 (+ 31us) server_negotiation.cc:407] Connection header received
0810 22:56:44.615430 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:44.615445 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:44.615828 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:44.616145 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:44.616922 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:44.616925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:44.616935 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:44.616944 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:44.616993 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:44.617006 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:44.617138 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.617200 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.617422 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":113,"threads_started":1}
I20260810 22:56:44.618579 2840 log.cc:824] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:44.620388 2840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7: No bootstrap required, opened a new log
I20260810 22:56:44.623797 2840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45695 } }
I20260810 22:56:44.623988 2840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:44.624054 2840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890e0eaa1bde4bdfbf841e9f4b369ff7, State: Initialized, Role: FOLLOWER
I20260810 22:56:44.624415 2840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45695 } }
I20260810 22:56:44.624544 2840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:44.624620 2840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:44.624720 2840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:44.625591 2840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45695 } }
I20260810 22:56:44.625985 2840 leader_election.cc:302] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890e0eaa1bde4bdfbf841e9f4b369ff7; no voters:
I20260810 22:56:44.626322 2840 leader_election.cc:288] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:44.626410 2845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:44.626618 2845 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [term 1 LEADER]: Becoming Leader. State: Replica: 890e0eaa1bde4bdfbf841e9f4b369ff7, State: Running, Role: LEADER
I20260810 22:56:44.626904 2845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45695 } }
I20260810 22:56:44.627408 2840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:44.628358 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890e0eaa1bde4bdfbf841e9f4b369ff7. Latest consensus state: current_term: 1 leader_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45695 } } }
I20260810 22:56:44.628472 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:44.628716 2846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0eaa1bde4bdfbf841e9f4b369ff7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45695 } } }
I20260810 22:56:44.628793 2846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:44.628875 2855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:44.631551 2855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:44.636271 2855 catalog_manager.cc:1347] Generated new cluster ID: 191b85561dcb4eb98a2cc8cbbc7f8d22
I20260810 22:56:44.636327 2855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:44.651978 2855 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:44.652269 2855 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:44.660343 2855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7: Generated new TSK 0
I20260810 22:56:44.660903 2855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:56:44.663854 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45695
--builtin_ntp_servers=127.29.78.148:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:56:44.777783 2869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:44.778087 2869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:44.778146 2869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:44.781993 2869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:44.782035 2869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:44.782114 2869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:44.786717 2869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:44.787642 2869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:44.788920 2869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:44.798285 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:44.798295 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:44.798454 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:44.798430 2869 server_base.cc:1034] running on GCE node
I20260810 22:56:44.798929 2869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:44.799489 2869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:44.800647 2869 hybrid_clock.cc:648] HybridClock initialized: now 1786402604800634 us; error 28 us; skew 500 ppm
I20260810 22:56:44.802239 2869 webserver.cc:460] Webserver started at http://127.29.78.129:35977/ using document root <none> and password file <none>
I20260810 22:56:44.802714 2869 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:44.802765 2869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:44.802907 2869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:44.804692 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "7fb291ecacf04d309f8517549d88e8a1"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "1503b8df0575bc75710e86a603db6544"
server_key_iv: "74fa43b7e85208bfa2d0719602311bdc"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.805222 2869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "7fb291ecacf04d309f8517549d88e8a1"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "1503b8df0575bc75710e86a603db6544"
server_key_iv: "74fa43b7e85208bfa2d0719602311bdc"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.808557 2869 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:56:44.810842 2884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:44.811990 2869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:44.812086 2869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "7fb291ecacf04d309f8517549d88e8a1"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "1503b8df0575bc75710e86a603db6544"
server_key_iv: "74fa43b7e85208bfa2d0719602311bdc"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.812166 2869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:44.816361 2869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:44.816838 2869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:44.816957 2869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:44.817382 2869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:44.818197 2869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:44.818241 2869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:44.818276 2869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:44.818291 2869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:44.826857 2869 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38561
I20260810 22:56:44.826900 3011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38561 every 8 connection(s)
I20260810 22:56:44.827693 2869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:44.833302 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.829414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:39383 (local address 127.29.78.190:45695)
0810 22:56:44.829552 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:44.829557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:44.830591 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 22:56:44.831831 (+ 1240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:44.831834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:44.831875 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:44.831949 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:44.832942 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:44.832944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:44.832946 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:44.832957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:44.832975 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:44.832982 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:44.833056 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:44.833163 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 22:56:44.833197 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:56:44.833763 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 2869
I20260810 22:56:44.833889 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:44.834112 30010 external_mini_cluster.cc:1428] Setting key 3f2992f52f5f965f5b24ac8c29f14f6e
I20260810 22:56:44.834100 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.829887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45695 (local address 127.29.78.129:39383)
0810 22:56:44.830447 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:44.830482 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:44.831620 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:44.832109 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:44.832115 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:44.832195 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:44.832685 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:44.832687 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:44.832722 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:44.832782 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:44.833057 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:44.833064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:44.833067 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:56:44.833180 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 22:56:44.833341 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260810 22:56:44.835206 3012 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45695
I20260810 22:56:44.835417 3012 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:44.835842 3012 heartbeater.cc:511] Master 127.29.78.190:45695 requested a full tablet report, sending...
I20260810 22:56:44.836203 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45695
--builtin_ntp_servers=127.29.78.148:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 22:56:44.837276 2801 ts_manager.cc:194] Registered new tserver with Master: 7fb291ecacf04d309f8517549d88e8a1 (127.29.78.129:38561)
W20260810 22:56:44.943126 3016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:44.943339 3016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:44.943373 3016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:44.947075 3016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:44.947117 3016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:44.947198 3016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:44.951342 3016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:44.952302 3016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:44.953588 3016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:44.962568 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:44.962597 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:44.962615 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:44.962970 3016 server_base.cc:1034] running on GCE node
I20260810 22:56:44.963341 3016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:44.963933 3016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:44.965143 3016 hybrid_clock.cc:648] HybridClock initialized: now 1786402604965134 us; error 28 us; skew 500 ppm
I20260810 22:56:44.967106 3016 webserver.cc:460] Webserver started at http://127.29.78.130:37829/ using document root <none> and password file <none>
I20260810 22:56:44.967741 3016 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:44.967829 3016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:44.968055 3016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:44.970118 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "373c5d2f1ec54df981eba7de2aebcec7"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "2de03ad2e97d0dc69f348d8f349c379d"
server_key_iv: "364d915818302b444be125b755f7fb5e"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.970767 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "373c5d2f1ec54df981eba7de2aebcec7"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "2de03ad2e97d0dc69f348d8f349c379d"
server_key_iv: "364d915818302b444be125b755f7fb5e"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.975215 3016 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:56:44.977589 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:44.978813 3016 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:56:44.978957 3016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "373c5d2f1ec54df981eba7de2aebcec7"
format_stamp: "Formatted at 2026-08-10 22:56:44 on dist-test-slave-6nmv"
server_key: "2de03ad2e97d0dc69f348d8f349c379d"
server_key_iv: "364d915818302b444be125b755f7fb5e"
server_key_version: "encryptionkey@0"
I20260810 22:56:44.979063 3016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:44.983724 3016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:44.984293 3016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:44.984467 3016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:44.984982 3016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:44.985941 3016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:44.985989 3016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:44.986052 3016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:44.986090 3016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:44.994621 3016 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:37207
I20260810 22:56:44.994652 3161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:37207 every 8 connection(s)
I20260810 22:56:44.995520 3016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:45.001219 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.997380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40663 (local address 127.29.78.190:45695)
0810 22:56:44.997532 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:44.997537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:44.998650 (+ 1113us) server_negotiation.cc:407] Connection header received
0810 22:56:44.999840 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:44.999842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:44.999881 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:44.999954 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.000888 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.000891 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.000892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.000903 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.000922 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.000929 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.000997 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.001081 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.001115 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:56:45.002041 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:44.997851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45695 (local address 127.29.78.130:40663)
0810 22:56:44.998495 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:44.998541 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:44.999664 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.000103 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.000109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.000186 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:45.000658 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:45.000660 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:45.000695 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:45.000756 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:45.000982 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:45.000985 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:45.000988 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:56:45.001091 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 22:56:45.001255 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
I20260810 22:56:45.002890 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3016
I20260810 22:56:45.003019 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:45.003141 3162 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45695
I20260810 22:56:45.003338 30010 external_mini_cluster.cc:1428] Setting key 07ca10f8c35727ecb51ea7a51eb61db7
I20260810 22:56:45.003351 3162 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:45.003746 3162 heartbeater.cc:511] Master 127.29.78.190:45695 requested a full tablet report, sending...
I20260810 22:56:45.004681 2801 ts_manager.cc:194] Registered new tserver with Master: 373c5d2f1ec54df981eba7de2aebcec7 (127.29.78.130:37207)
I20260810 22:56:45.005290 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45695
--builtin_ntp_servers=127.29.78.148:33001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 22:56:45.113283 3165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:45.113528 3165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:45.113564 3165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:45.117738 3165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:45.117789 3165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:45.117882 3165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:45.122189 3165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:33001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:45695
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:45.123123 3165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:45.124361 3165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:45.133868 3170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:45.133917 3171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:45.133917 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:45.134157 3165 server_base.cc:1034] running on GCE node
I20260810 22:56:45.134618 3165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:45.135277 3165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:45.136430 3165 hybrid_clock.cc:648] HybridClock initialized: now 1786402605136416 us; error 28 us; skew 500 ppm
I20260810 22:56:45.138211 3165 webserver.cc:460] Webserver started at http://127.29.78.131:34131/ using document root <none> and password file <none>
I20260810 22:56:45.138823 3165 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:45.138892 3165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:45.139140 3165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:45.141088 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "cd8ca1be65d44382aa4142a2356e0c11"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "520b44c25eb608b707972df6fe007629"
server_key_iv: "95de4e9f100e6f1a51dceb0ee948021d"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.141613 3165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "cd8ca1be65d44382aa4142a2356e0c11"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "520b44c25eb608b707972df6fe007629"
server_key_iv: "95de4e9f100e6f1a51dceb0ee948021d"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.144925 3165 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:45.147279 3180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:45.148388 3165 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 22:56:45.148514 3165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "cd8ca1be65d44382aa4142a2356e0c11"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "520b44c25eb608b707972df6fe007629"
server_key_iv: "95de4e9f100e6f1a51dceb0ee948021d"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.148617 3165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:45.153117 3165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:45.153687 3165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:45.153935 3165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:45.154471 3165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:45.155427 3165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:45.155498 3165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:45.155567 3165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:45.155611 3165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:45.165153 3165 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:43147
I20260810 22:56:45.165172 3304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:43147 every 8 connection(s)
I20260810 22:56:45.166113 3165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:45.171568 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3165
I20260810 22:56:45.171733 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:45.171999 30010 external_mini_cluster.cc:1428] Setting key 78216ee8749c229d2dbd07dcd42a5c03
I20260810 22:56:45.172013 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.167903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:57417 (local address 127.29.78.190:45695)
0810 22:56:45.168068 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.168073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.169100 (+ 1027us) server_negotiation.cc:407] Connection header received
0810 22:56:45.170400 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.170402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.170438 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.170510 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.171488 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.171491 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.171494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.171515 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.171544 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.171552 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.171684 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.171826 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.171875 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:56:45.173099 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.168344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45695 (local address 127.29.78.131:57417)
0810 22:56:45.168949 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.168988 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.170216 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.170660 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.170666 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.170747 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:45.171240 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:45.171242 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:45.171279 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:45.171341 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:45.171667 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:45.171675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:45.171681 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:56:45.171866 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 22:56:45.172104 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":100,"threads_started":1}
I20260810 22:56:45.174225 3305 heartbeater.cc:347] Connected to a master server at 127.29.78.190:45695
I20260810 22:56:45.174476 3305 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:45.174877 3305 heartbeater.cc:511] Master 127.29.78.190:45695 requested a full tablet report, sending...
I20260810 22:56:45.175853 2801 ts_manager.cc:194] Registered new tserver with Master: cd8ca1be65d44382aa4142a2356e0c11 (127.29.78.131:43147)
I20260810 22:56:45.186031 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:56:45.189507 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.187892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.29.78.190:45695)
0810 22:56:45.188054 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.188060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.188203 (+ 143us) server_negotiation.cc:407] Connection header received
0810 22:56:45.188350 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.188353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.188388 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.188476 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.189115 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.189119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.189121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.189132 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.189152 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.189163 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.189254 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.189330 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.189369 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:56:45.191980 2801 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:45.195077 2801 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:45.206459 3320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.203428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:43147 (local address 127.0.0.1:46178)
0810 22:56:45.203892 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.203940 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.204044 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.204920 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.204925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.205002 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:45.205600 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:45.205604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:45.205626 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:45.205693 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:45.206109 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:45.206118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:45.206123 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 22:56:45.206272 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 22:56:45.206335 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"spinlock_wait_cycles":6528,"thread_start_us":98,"threads_started":1}
I20260810 22:56:45.206476 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.203987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.29.78.131:43147)
0810 22:56:45.204352 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.204362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.204385 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:56:45.204433 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.204439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.204763 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.205298 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.205933 (+ 635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.205941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.205954 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.205968 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.206010 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.206036 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.206125 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.206295 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.206360 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":154,"threads_started":1}
I20260810 22:56:45.206931 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.203233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38561 (local address 127.0.0.1:34516)
0810 22:56:45.203894 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.203940 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.204044 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.205743 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.205746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.205761 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:45.206285 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:45.206287 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:45.206297 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:45.206320 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:45.206700 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:45.206703 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:45.206706 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:56:45.206765 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 22:56:45.206797 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"spinlock_wait_cycles":6784,"thread_start_us":75,"threads_started":1}
I20260810 22:56:45.207252 3323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.203640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37207 (local address 127.0.0.1:53550)
0810 22:56:45.204573 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.204585 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.204661 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.205172 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.205178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.205195 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:45.205865 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:45.205867 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:45.205877 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:45.205900 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:45.206846 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:45.206849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:45.206852 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:56:45.206933 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:56:45.206965 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":78,"threads_started":1}
I20260810 22:56:45.207401 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.203412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.29.78.129:38561)
0810 22:56:45.204295 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.204307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.204331 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:56:45.204397 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.204403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.204591 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.204721 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.206470 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.206478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.206489 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.206501 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.206539 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.206576 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.206664 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.207156 (+ 492us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.207238 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":116,"threads_started":1}
I20260810 22:56:45.207854 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.204033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53550 (local address 127.29.78.130:37207)
0810 22:56:45.204430 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.204439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.204748 (+ 309us) server_negotiation.cc:407] Connection header received
0810 22:56:45.204803 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.204807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.204986 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.205150 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.206317 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.206324 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.206334 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.206347 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.206396 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.206431 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.206808 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.207655 (+ 847us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.207721 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":136,"threads_started":1}
I20260810 22:56:45.209270 3231 tablet_service.cc:1467] Processing CreateTablet for tablet 193e3e37b2ad4b0493bd4da948ac4743 (DEFAULT_TABLE table=test-table [id=d0846ce7a1254271a22e5ba242345a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:45.209882 2940 tablet_service.cc:1467] Processing CreateTablet for tablet 193e3e37b2ad4b0493bd4da948ac4743 (DEFAULT_TABLE table=test-table [id=d0846ce7a1254271a22e5ba242345a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:45.210311 3231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193e3e37b2ad4b0493bd4da948ac4743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:45.210302 3087 tablet_service.cc:1467] Processing CreateTablet for tablet 193e3e37b2ad4b0493bd4da948ac4743 (DEFAULT_TABLE table=test-table [id=d0846ce7a1254271a22e5ba242345a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:45.210875 2940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193e3e37b2ad4b0493bd4da948ac4743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:45.211231 3087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193e3e37b2ad4b0493bd4da948ac4743. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:45.216619 3326 tablet_bootstrap.cc:492] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Bootstrap starting.
I20260810 22:56:45.218153 3327 tablet_bootstrap.cc:492] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Bootstrap starting.
I20260810 22:56:45.218799 3328 tablet_bootstrap.cc:492] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1: Bootstrap starting.
I20260810 22:56:45.219306 3326 tablet_bootstrap.cc:654] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:45.220175 3326 log.cc:824] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:45.220263 3327 tablet_bootstrap.cc:654] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:45.220664 3328 tablet_bootstrap.cc:654] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:45.221071 3327 log.cc:824] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:45.221519 3328 log.cc:824] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:45.222332 3326 tablet_bootstrap.cc:492] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: No bootstrap required, opened a new log
I20260810 22:56:45.222525 3326 ts_tablet_manager.cc:1397] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:56:45.222995 3327 tablet_bootstrap.cc:492] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: No bootstrap required, opened a new log
I20260810 22:56:45.223165 3327 ts_tablet_manager.cc:1397] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:56:45.223237 3328 tablet_bootstrap.cc:492] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1: No bootstrap required, opened a new log
I20260810 22:56:45.223404 3328 ts_tablet_manager.cc:1397] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 22:56:45.226886 3328 raft_consensus.cc:348] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.226915 3326 raft_consensus.cc:348] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.227121 3328 raft_consensus.cc:374] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:45.227162 3326 raft_consensus.cc:374] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:45.227170 3328 raft_consensus.cc:729] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fb291ecacf04d309f8517549d88e8a1, State: Initialized, Role: FOLLOWER
I20260810 22:56:45.227213 3326 raft_consensus.cc:729] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8ca1be65d44382aa4142a2356e0c11, State: Initialized, Role: FOLLOWER
I20260810 22:56:45.227341 3327 raft_consensus.cc:348] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.227595 3327 raft_consensus.cc:374] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:45.227648 3327 raft_consensus.cc:729] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 373c5d2f1ec54df981eba7de2aebcec7, State: Initialized, Role: FOLLOWER
I20260810 22:56:45.227662 3328 consensus_queue.cc:260] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.227730 3326 consensus_queue.cc:260] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.228029 3327 consensus_queue.cc:260] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.228372 3012 heartbeater.cc:503] Master 127.29.78.190:45695 was elected leader, sending a full tablet report...
I20260810 22:56:45.228715 3328 ts_tablet_manager.cc:1428] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260810 22:56:45.228753 3326 ts_tablet_manager.cc:1428] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 22:56:45.228807 3327 ts_tablet_manager.cc:1428] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260810 22:56:45.229055 3162 heartbeater.cc:503] Master 127.29.78.190:45695 was elected leader, sending a full tablet report...
I20260810 22:56:45.229979 3305 heartbeater.cc:503] Master 127.29.78.190:45695 was elected leader, sending a full tablet report...
W20260810 22:56:45.246546 3163 tablet.cc:2369] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:45.247815 3332 raft_consensus.cc:482] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:45.247990 3332 raft_consensus.cc:504] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.248982 3332 leader_election.cc:288] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd8ca1be65d44382aa4142a2356e0c11 (127.29.78.131:43147), 373c5d2f1ec54df981eba7de2aebcec7 (127.29.78.130:37207)
I20260810 22:56:45.250916 3015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.249225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37207 (local address 127.29.78.129:36283)
0810 22:56:45.249346 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.249362 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.249440 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.249747 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.249752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.249771 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:45.250330 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:45.250332 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:45.250375 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:45.250415 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:45.250674 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:45.250677 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:45.250681 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:56:45.250759 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 22:56:45.250795 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 22:56:45.250916 3325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.249298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:36283 (local address 127.29.78.130:37207)
0810 22:56:45.249423 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.249428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.249445 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:56:45.249536 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.249541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.249577 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.249678 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.250524 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.250527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.250530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.250542 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.250570 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.250583 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.250678 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.250756 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.250795 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:56:45.251435 3324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.249325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:35683 (local address 127.29.78.131:43147)
0810 22:56:45.249444 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.249448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.249910 (+ 462us) server_negotiation.cc:407] Connection header received
0810 22:56:45.250063 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.250067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.250103 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.250190 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.250975 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.250979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.250981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.250997 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.251021 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.251032 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.251118 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.251258 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.251318 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:45.251497 3335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.249225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:43147 (local address 127.29.78.129:35683)
0810 22:56:45.249811 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.249825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.249924 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.250244 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.250250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.250266 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:45.250794 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:56:45.250796 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:45.250806 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:56:45.250829 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:45.251161 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:45.251165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:45.251168 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:56:45.251248 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:56:45.251286 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":150,"thread_start_us":124,"threads_started":1}
I20260810 22:56:45.251560 3107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193e3e37b2ad4b0493bd4da948ac4743" candidate_uuid: "7fb291ecacf04d309f8517549d88e8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373c5d2f1ec54df981eba7de2aebcec7" is_pre_election: true
I20260810 22:56:45.251874 3107 raft_consensus.cc:2454] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb291ecacf04d309f8517549d88e8a1 in term 0.
I20260810 22:56:45.252180 3252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193e3e37b2ad4b0493bd4da948ac4743" candidate_uuid: "7fb291ecacf04d309f8517549d88e8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8ca1be65d44382aa4142a2356e0c11" is_pre_election: true
I20260810 22:56:45.252336 2887 leader_election.cc:302] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 373c5d2f1ec54df981eba7de2aebcec7, 7fb291ecacf04d309f8517549d88e8a1; no voters:
I20260810 22:56:45.252480 3252 raft_consensus.cc:2454] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7fb291ecacf04d309f8517549d88e8a1 in term 0.
I20260810 22:56:45.252596 3332 raft_consensus.cc:2781] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:45.252679 3332 raft_consensus.cc:482] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:45.252748 3332 raft_consensus.cc:3037] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:45.253852 3332 raft_consensus.cc:504] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.254201 3332 leader_election.cc:288] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [CANDIDATE]: Term 1 election: Requested vote from peers cd8ca1be65d44382aa4142a2356e0c11 (127.29.78.131:43147), 373c5d2f1ec54df981eba7de2aebcec7 (127.29.78.130:37207)
I20260810 22:56:45.254529 3107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193e3e37b2ad4b0493bd4da948ac4743" candidate_uuid: "7fb291ecacf04d309f8517549d88e8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "373c5d2f1ec54df981eba7de2aebcec7"
I20260810 22:56:45.254515 3252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193e3e37b2ad4b0493bd4da948ac4743" candidate_uuid: "7fb291ecacf04d309f8517549d88e8a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8ca1be65d44382aa4142a2356e0c11"
I20260810 22:56:45.254635 3252 raft_consensus.cc:3037] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:45.254634 3107 raft_consensus.cc:3037] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:45.255888 3107 raft_consensus.cc:2454] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb291ecacf04d309f8517549d88e8a1 in term 1.
I20260810 22:56:45.255883 3252 raft_consensus.cc:2454] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7fb291ecacf04d309f8517549d88e8a1 in term 1.
I20260810 22:56:45.256155 2885 leader_election.cc:302] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7fb291ecacf04d309f8517549d88e8a1, cd8ca1be65d44382aa4142a2356e0c11; no voters:
I20260810 22:56:45.256350 3332 raft_consensus.cc:2781] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:45.256561 3332 raft_consensus.cc:686] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 1 LEADER]: Becoming Leader. State: Replica: 7fb291ecacf04d309f8517549d88e8a1, State: Running, Role: LEADER
I20260810 22:56:45.256830 3332 consensus_queue.cc:237] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } }
I20260810 22:56:45.259585 2799 catalog_manager.cc:5495] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7fb291ecacf04d309f8517549d88e8a1 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "7fb291ecacf04d309f8517549d88e8a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7fb291ecacf04d309f8517549d88e8a1" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:45.267791 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.266584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34530 (local address 127.29.78.129:38561)
0810 22:56:45.266687 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.266690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.266702 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:56:45.266809 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.266811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.266854 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.266934 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.267524 (+ 590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.267526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.267528 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.267534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.267547 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.267553 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.267643 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.267686 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.267713 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 22:56:45.273046 3107 raft_consensus.cc:1261] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 1 FOLLOWER]: Refusing update from remote peer 7fb291ecacf04d309f8517549d88e8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:45.273046 3252 raft_consensus.cc:1261] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 1 FOLLOWER]: Refusing update from remote peer 7fb291ecacf04d309f8517549d88e8a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:45.273764 3336 consensus_queue.cc:1035] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8ca1be65d44382aa4142a2356e0c11" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 43147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:45.274001 3332 consensus_queue.cc:1035] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "373c5d2f1ec54df981eba7de2aebcec7" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:45.288918 2799 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:45.289086 2799 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:45.291371 2799 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 890e0eaa1bde4bdfbf841e9f4b369ff7: Sending DeleteTablet for 3 replicas of tablet 193e3e37b2ad4b0493bd4da948ac4743
I20260810 22:56:45.292106 2940 tablet_service.cc:1514] Processing DeleteTablet for tablet 193e3e37b2ad4b0493bd4da948ac4743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:45 UTC) from {username='slave'} at 127.0.0.1:34516
I20260810 22:56:45.292107 3231 tablet_service.cc:1514] Processing DeleteTablet for tablet 193e3e37b2ad4b0493bd4da948ac4743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:45 UTC) from {username='slave'} at 127.0.0.1:46178
I20260810 22:56:45.292397 3087 tablet_service.cc:1514] Processing DeleteTablet for tablet 193e3e37b2ad4b0493bd4da948ac4743 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:45 UTC) from {username='slave'} at 127.0.0.1:53550
I20260810 22:56:45.292559 3349 tablet_replica.cc:330] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1: stopping tablet replica
I20260810 22:56:45.292912 3349 raft_consensus.cc:2229] T 193e3e37b2ad4b0493bd4da948ac4743 P 7fb291ecacf04d309f8517549d88e8a1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:45.292873 3351 tablet_replica.cc:330] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: stopping tablet replica
I20260810 22:56:45.293339 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 2869
I20260810 22:56:45.295718 3350 tablet_replica.cc:330] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: stopping tablet replica
I20260810 22:56:45.296128 3350 raft_consensus.cc:2229] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:45.296423 3350 raft_consensus.cc:2258] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:45.297253 3351 raft_consensus.cc:2229] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:45.297526 3351 raft_consensus.cc:2258] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:45.298002 3350 ts_tablet_manager.cc:1905] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:45.298905 3351 ts_tablet_manager.cc:1905] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:45.300917 3350 ts_tablet_manager.cc:1918] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:45.301011 3350 log.cc:1196] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/193e3e37b2ad4b0493bd4da948ac4743
I20260810 22:56:45.301307 3350 ts_tablet_manager.cc:1939] T 193e3e37b2ad4b0493bd4da948ac4743 P cd8ca1be65d44382aa4142a2356e0c11: Deleting consensus metadata
I20260810 22:56:45.301780 3351 ts_tablet_manager.cc:1918] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:45.301950 3351 log.cc:1196] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/193e3e37b2ad4b0493bd4da948ac4743
I20260810 22:56:45.302287 3351 ts_tablet_manager.cc:1939] T 193e3e37b2ad4b0493bd4da948ac4743 P 373c5d2f1ec54df981eba7de2aebcec7: Deleting consensus metadata
I20260810 22:56:45.302701 2784 catalog_manager.cc:4841] TS cd8ca1be65d44382aa4142a2356e0c11 (127.29.78.131:43147): tablet 193e3e37b2ad4b0493bd4da948ac4743 (table test-table [id=d0846ce7a1254271a22e5ba242345a2c]) successfully deleted
W20260810 22:56:45.303207 2785 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:38561 (error 108)
I20260810 22:56:45.303357 2786 catalog_manager.cc:4841] TS 373c5d2f1ec54df981eba7de2aebcec7 (127.29.78.130:37207): tablet 193e3e37b2ad4b0493bd4da948ac4743 (table test-table [id=d0846ce7a1254271a22e5ba242345a2c]) successfully deleted
I20260810 22:56:45.303398 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3016
I20260810 22:56:45.304375 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.304053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38561 (local address 127.0.0.1:34542)
0810 22:56:45.304204 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.304287 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 22:56:45.304524 2785 catalog_manager.cc:4584] TS 7fb291ecacf04d309f8517549d88e8a1 (127.29.78.129:38561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 193e3e37b2ad4b0493bd4da948ac4743: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38561: connect: Connection refused (error 111)
I20260810 22:56:45.311373 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3165
I20260810 22:56:45.319131 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 2768
2026-08-10T22:56:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (901 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 22:56:45.341771 30010 mini_oidc.cc:152] Starting JWKS server
I20260810 22:56:45.343461 30010 webserver.cc:460] Webserver started at https://127.0.0.1:42599/ using document root <none> and password file <none>
I20260810 22:56:45.343642 30010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:56:45.343921 30010 webserver.cc:460] Webserver started at http://127.0.0.1:36007/ using document root <none> and password file <none>
2026-08-10T22:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:45Z Disabled control of system clock
I20260810 22:56:45.353446 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43097
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43097
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42599/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/testchainca.cert with env {}
W20260810 22:56:45.462611 3359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:45.462860 3359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:45.462894 3359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:45.466699 3359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:45.466747 3359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:45.466763 3359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:45.466778 3359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:45.466794 3359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:56:45.466807 3359 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:56:45.466832 3359 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42599/jwks.json
I20260810 22:56:45.471765 3359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42599/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:45.472688 3359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:45.473627 3359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:45.481536 3365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:45.481602 3364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:45.481536 3368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:45.481678 3359 server_base.cc:1034] running on GCE node
I20260810 22:56:45.482343 3359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:45.483342 3359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:45.484531 3359 hybrid_clock.cc:648] HybridClock initialized: now 1786402605484517 us; error 35 us; skew 500 ppm
I20260810 22:56:45.486400 3359 webserver.cc:460] Webserver started at http://127.29.78.190:34455/ using document root <none> and password file <none>
I20260810 22:56:45.486914 3359 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:45.487001 3359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:45.487210 3359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:45.489162 3359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "fc1441125eca4af2a2e54919ce780b0c"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "9c88e4245cdf3480488c81f617ee63ff"
server_key_iv: "5b23635428aee08f9d52488c5f1a527f"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.489719 3359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "fc1441125eca4af2a2e54919ce780b0c"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "9c88e4245cdf3480488c81f617ee63ff"
server_key_iv: "5b23635428aee08f9d52488c5f1a527f"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.493058 3359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:45.495385 3377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:45.496552 3359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 22:56:45.496692 3359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "fc1441125eca4af2a2e54919ce780b0c"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "9c88e4245cdf3480488c81f617ee63ff"
server_key_iv: "5b23635428aee08f9d52488c5f1a527f"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.496793 3359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:45.505952 3359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:45.506512 3359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:45.506687 3359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:45.513119 3359 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43097
I20260810 22:56:45.513159 3435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43097 every 8 connection(s)
I20260810 22:56:45.514060 3359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:45.516948 3436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:45.518513 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3359
I20260810 22:56:45.518643 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:45.518889 30010 external_mini_cluster.cc:1428] Setting key b6a2ce0e76f51eaa62a6abdc3dc449d5
I20260810 22:56:45.522683 3436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c: Bootstrap starting.
I20260810 22:56:45.524753 3436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:45.525604 3436 log.cc:824] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:45.527279 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.520106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.29.78.190:43097)
0810 22:56:45.520659 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.520672 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.520704 (+ 32us) server_negotiation.cc:407] Connection header received
0810 22:56:45.521304 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.521319 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.521654 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.521994 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.522365 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.523274 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.524144 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.524570 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.524750 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:45.525604 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.525609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.525621 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.526092 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.526145 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.526165 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.526283 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.526425 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.526679 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
I20260810 22:56:45.527563 3436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c: No bootstrap required, opened a new log
I20260810 22:56:45.529767 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:56:45.531093 3436 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1441125eca4af2a2e54919ce780b0c" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43097 } }
I20260810 22:56:45.531373 3436 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:45.531435 3436 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1441125eca4af2a2e54919ce780b0c, State: Initialized, Role: FOLLOWER
I20260810 22:56:45.531910 3436 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1441125eca4af2a2e54919ce780b0c" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43097 } }
I20260810 22:56:45.532056 3436 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:45.532159 3436 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:45.532258 3436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:45.533106 3436 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1441125eca4af2a2e54919ce780b0c" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43097 } }
I20260810 22:56:45.533411 3436 leader_election.cc:302] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc1441125eca4af2a2e54919ce780b0c; no voters:
I20260810 22:56:45.533738 3436 leader_election.cc:288] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:45.533800 3441 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:45.534015 3441 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [term 1 LEADER]: Becoming Leader. State: Replica: fc1441125eca4af2a2e54919ce780b0c, State: Running, Role: LEADER
I20260810 22:56:45.534267 3441 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1441125eca4af2a2e54919ce780b0c" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43097 } }
I20260810 22:56:45.534876 3436 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:45.535926 3442 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc1441125eca4af2a2e54919ce780b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1441125eca4af2a2e54919ce780b0c" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43097 } } }
I20260810 22:56:45.535950 3443 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc1441125eca4af2a2e54919ce780b0c. Latest consensus state: current_term: 1 leader_uuid: "fc1441125eca4af2a2e54919ce780b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1441125eca4af2a2e54919ce780b0c" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43097 } } }
I20260810 22:56:45.536060 3442 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:45.536070 3443 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:45.537220 3451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:45.539623 3451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:45.544399 3451 catalog_manager.cc:1347] Generated new cluster ID: 0dca730ee29242b491f4c490880b3e88
I20260810 22:56:45.544467 3451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:45.553054 3451 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:45.554195 3451 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:45.560318 3451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc1441125eca4af2a2e54919ce780b0c: Generated new TSK 0
I20260810 22:56:45.560943 3451 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 22:56:45.589118 3470 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 22:56:45.589237 3470 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 22:56:45.589838 3439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.573456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.29.78.190:43097)
0810 22:56:45.573596 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.573601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.573794 (+ 193us) server_negotiation.cc:407] Connection header received
0810 22:56:45.574031 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.574034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.574088 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.574195 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:56:45.574420 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.575279 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.576096 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.576472 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.576572 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:45.576854 (+ 282us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:56:45.589415 (+ 12561us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 22:56:45.589672 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 22:56:45.589905 3439 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 22:56:45.591320 3469 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:45.573325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43097 (local address 127.0.0.1:58142)
0810 22:56:45.573714 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.573726 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.573864 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.574212 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.574214 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.574220 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:56:45.574321 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:45.574326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.575378 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.575381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:45.575989 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:45.575996 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.576581 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.576584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:45.576650 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:45.576690 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:56:45.589634 (+ 12944us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 22:56:45.589869 (+ 235us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:43097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.29.78.190:43097: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42599/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 22:56:45.592394 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3359
2026-08-10T22:56:45Z chronyd exiting
I20260810 22:56:45.632450 30010 test_util.cc:175] -----------------------------------------------
I20260810 22:56:45.632535 30010 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402578782454-30010-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (292 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 22:56:45.633584 30010 mini_oidc.cc:152] Starting JWKS server
I20260810 22:56:45.634418 30010 webserver.cc:460] Webserver started at https://127.0.0.1:38381/ using document root <none> and password file <none>
I20260810 22:56:45.634493 30010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:56:45.634680 30010 webserver.cc:460] Webserver started at http://127.0.0.1:35919/ using document root <none> and password file <none>
2026-08-10T22:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:45Z Disabled control of system clock
I20260810 22:56:45.644197 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36385
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36385
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38381/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/testchainca.cert with env {}
W20260810 22:56:45.748181 3476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:45.748449 3476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:45.748486 3476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:45.752552 3476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:45.752610 3476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:45.752626 3476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:45.752643 3476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:45.752658 3476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:56:45.752674 3476 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:56:45.752693 3476 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38381/jwks.json
I20260810 22:56:45.757627 3476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36385
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38381/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:45.758646 3476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:45.759554 3476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:45.767854 3481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:45.767848 3482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:45.767879 3476 server_base.cc:1034] running on GCE node
W20260810 22:56:45.767849 3484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:45.768549 3476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:45.769438 3476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:45.770614 3476 hybrid_clock.cc:648] HybridClock initialized: now 1786402605770601 us; error 33 us; skew 500 ppm
I20260810 22:56:45.772305 3476 webserver.cc:460] Webserver started at http://127.29.78.190:38361/ using document root <none> and password file <none>
I20260810 22:56:45.772852 3476 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:45.772961 3476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:45.773180 3476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:45.775315 3476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "625adf92ac054e19baab7a99007b2045"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "fc9f4acd4790cde5a01f6958cc79b103"
server_key_iv: "3fb3a2203fe0931ab9d9161c6fe0f249"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.775882 3476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "625adf92ac054e19baab7a99007b2045"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "fc9f4acd4790cde5a01f6958cc79b103"
server_key_iv: "3fb3a2203fe0931ab9d9161c6fe0f249"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.779489 3476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:45.781674 3491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:45.782862 3476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:45.783007 3476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "625adf92ac054e19baab7a99007b2045"
format_stamp: "Formatted at 2026-08-10 22:56:45 on dist-test-slave-6nmv"
server_key: "fc9f4acd4790cde5a01f6958cc79b103"
server_key_iv: "3fb3a2203fe0931ab9d9161c6fe0f249"
server_key_version: "encryptionkey@0"
I20260810 22:56:45.783109 3476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:45.793497 3476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:45.794127 3476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:45.794301 3476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:45.800734 3476 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36385
I20260810 22:56:45.800740 3551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36385 every 8 connection(s)
I20260810 22:56:45.801524 3476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:45.804435 3552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:45.809106 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3476
I20260810 22:56:45.809248 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:45.809557 30010 external_mini_cluster.cc:1428] Setting key d6b560e76dbae7cf8a354372e6539b29
I20260810 22:56:45.809887 3552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045: Bootstrap starting.
I20260810 22:56:45.812340 3552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:45.813131 3552 log.cc:824] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:45.814947 3552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045: No bootstrap required, opened a new log
I20260810 22:56:45.818516 3555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.810819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.29.78.190:36385)
0810 22:56:45.811424 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.811438 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.811471 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:56:45.812090 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.812105 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.812535 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.812897 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:45.813402 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.814497 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.815401 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.815872 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.816060 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:45.816929 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:45.816934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:45.816945 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:45.817280 (+ 335us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:45.817327 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:45.817344 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:45.817448 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:45.817588 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 22:56:45.817868 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":138,"threads_started":1}
I20260810 22:56:45.818447 3552 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625adf92ac054e19baab7a99007b2045" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36385 } }
I20260810 22:56:45.818675 3552 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:45.818738 3552 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625adf92ac054e19baab7a99007b2045, State: Initialized, Role: FOLLOWER
I20260810 22:56:45.819139 3552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625adf92ac054e19baab7a99007b2045" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36385 } }
I20260810 22:56:45.819264 3552 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:45.819314 3552 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:45.819386 3552 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:45.820220 3552 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625adf92ac054e19baab7a99007b2045" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36385 } }
I20260810 22:56:45.820569 3552 leader_election.cc:302] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625adf92ac054e19baab7a99007b2045; no voters:
I20260810 22:56:45.820896 3552 leader_election.cc:288] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:45.820957 3558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:45.821203 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:56:45.821228 3558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [term 1 LEADER]: Becoming Leader. State: Replica: 625adf92ac054e19baab7a99007b2045, State: Running, Role: LEADER
I20260810 22:56:45.821499 3558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625adf92ac054e19baab7a99007b2045" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36385 } }
I20260810 22:56:45.822081 3552 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:45.822985 3559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625adf92ac054e19baab7a99007b2045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625adf92ac054e19baab7a99007b2045" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36385 } } }
I20260810 22:56:45.823107 3559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:45.823460 3568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:45.823421 3560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625adf92ac054e19baab7a99007b2045. Latest consensus state: current_term: 1 leader_uuid: "625adf92ac054e19baab7a99007b2045" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625adf92ac054e19baab7a99007b2045" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36385 } } }
I20260810 22:56:45.824167 3560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:45.826380 3568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:45.831790 3568 catalog_manager.cc:1347] Generated new cluster ID: 26e381f596f344fa95f3b8f588b2b598
I20260810 22:56:45.831861 3568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:45.846441 3568 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:45.847240 3568 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:45.855716 3568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625adf92ac054e19baab7a99007b2045: Generated new TSK 0
I20260810 22:56:45.856361 3568 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 22:56:45.884349 3586 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 22:56:45.884455 3586 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 22:56:45.885210 3555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:45.867747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.29.78.190:36385)
0810 22:56:45.867866 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:45.867870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:45.868018 (+ 148us) server_negotiation.cc:407] Connection header received
0810 22:56:45.868138 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:45.868141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:45.868189 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:45.868296 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:56:45.868471 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.869238 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.870064 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.870377 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.870471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:45.870695 (+ 224us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:56:45.884734 (+ 14039us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 22:56:45.885032 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 22:56:45.885279 3555 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39560: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 22:56:45.885366 3585 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:45.867643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36385 (local address 127.0.0.1:39560)
0810 22:56:45.867941 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:45.867953 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:45.868039 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:45.868291 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:45.868293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:45.868299 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:56:45.868380 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:45.868383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.869332 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.869334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:45.869961 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:45.869970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:45.870481 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:45.870484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:45.870541 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:45.870560 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:56:45.884991 (+ 14431us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 22:56:45.885181 (+ 190us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:36385: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38381/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":107,"threads_started":1}
I20260810 22:56:45.886408 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3476
2026-08-10T22:56:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 22:56:45.928855 30010 mini_oidc.cc:152] Starting JWKS server
I20260810 22:56:45.929718 30010 webserver.cc:460] Webserver started at https://127.0.0.1:36827/ using document root <none> and password file <none>
I20260810 22:56:45.929869 30010 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 22:56:45.930163 30010 webserver.cc:460] Webserver started at http://127.0.0.1:39519/ using document root <none> and password file <none>
2026-08-10T22:56:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:45Z Disabled control of system clock
I20260810 22:56:45.940598 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35335
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35335
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36827/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 22:56:46.048565 3592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:46.048874 3592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:46.048978 3592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:46.052798 3592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:46.052901 3592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:46.052932 3592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:46.052973 3592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:46.053006 3592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:56:46.053045 3592 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:56:46.053079 3592 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36827/jwks.json
I20260810 22:56:46.057593 3592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36827/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:46.058590 3592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:46.059872 3592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:46.067843 3597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:46.067867 3600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:46.067881 3598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:46.068099 3592 server_base.cc:1034] running on GCE node
I20260810 22:56:46.068598 3592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:46.069555 3592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:46.070739 3592 hybrid_clock.cc:648] HybridClock initialized: now 1786402606070691 us; error 57 us; skew 500 ppm
I20260810 22:56:46.072626 3592 webserver.cc:460] Webserver started at http://127.29.78.190:35979/ using document root <none> and password file <none>
I20260810 22:56:46.073185 3592 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:46.073246 3592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:46.073449 3592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:46.075606 3592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "505f973bc0bd4a11ad89ffe96ed01a61"
format_stamp: "Formatted at 2026-08-10 22:56:46 on dist-test-slave-6nmv"
server_key: "c6572758cc150285081423603f31c794"
server_key_iv: "1f6faabee0db53bc905557abbaeace63"
server_key_version: "encryptionkey@0"
I20260810 22:56:46.076180 3592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "505f973bc0bd4a11ad89ffe96ed01a61"
format_stamp: "Formatted at 2026-08-10 22:56:46 on dist-test-slave-6nmv"
server_key: "c6572758cc150285081423603f31c794"
server_key_iv: "1f6faabee0db53bc905557abbaeace63"
server_key_version: "encryptionkey@0"
I20260810 22:56:46.079857 3592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:46.082221 3608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:46.083386 3592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:46.083520 3592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "505f973bc0bd4a11ad89ffe96ed01a61"
format_stamp: "Formatted at 2026-08-10 22:56:46 on dist-test-slave-6nmv"
server_key: "c6572758cc150285081423603f31c794"
server_key_iv: "1f6faabee0db53bc905557abbaeace63"
server_key_version: "encryptionkey@0"
I20260810 22:56:46.083611 3592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:46.096854 3592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:46.097419 3592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:46.097582 3592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:46.103742 3592 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35335
I20260810 22:56:46.103747 3664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35335 every 8 connection(s)
I20260810 22:56:46.104532 3592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:46.106084 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3592
I20260810 22:56:46.106211 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:46.106534 30010 external_mini_cluster.cc:1428] Setting key ec7d0d72e63f28af223e094a151bedbe
I20260810 22:56:46.108022 3665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:46.114703 3665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61: Bootstrap starting.
I20260810 22:56:46.116472 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:46.107823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.29.78.190:35335)
0810 22:56:46.108510 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:46.108530 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:46.108573 (+ 43us) server_negotiation.cc:407] Connection header received
0810 22:56:46.109430 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:46.109452 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:46.109962 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:46.110310 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:46.110745 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.112117 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.112952 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.113574 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.113731 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:46.114511 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:46.114516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:46.114529 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:46.114946 (+ 417us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:46.115009 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:46.115032 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:46.115115 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:46.115268 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 22:56:46.115528 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":148,"threads_started":1}
I20260810 22:56:46.117347 3665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:46.118191 3665 log.cc:824] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:46.119130 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:56:46.120352 3665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61: No bootstrap required, opened a new log
I20260810 22:56:46.123574 3665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35335 } }
I20260810 22:56:46.123799 3665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:46.123883 3665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 505f973bc0bd4a11ad89ffe96ed01a61, State: Initialized, Role: FOLLOWER
I20260810 22:56:46.124305 3665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35335 } }
I20260810 22:56:46.124442 3665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:46.124505 3665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:46.124606 3665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:46.125447 3665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35335 } }
I20260810 22:56:46.125778 3665 leader_election.cc:302] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 505f973bc0bd4a11ad89ffe96ed01a61; no voters:
I20260810 22:56:46.126147 3665 leader_election.cc:288] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:46.126263 3671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:46.126495 3671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [term 1 LEADER]: Becoming Leader. State: Replica: 505f973bc0bd4a11ad89ffe96ed01a61, State: Running, Role: LEADER
I20260810 22:56:46.126773 3671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35335 } }
I20260810 22:56:46.127292 3665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:46.127703 3673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 505f973bc0bd4a11ad89ffe96ed01a61. Latest consensus state: current_term: 1 leader_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35335 } } }
I20260810 22:56:46.127820 3673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:46.128139 3672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "505f973bc0bd4a11ad89ffe96ed01a61" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35335 } } }
I20260810 22:56:46.128239 3672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:46.128198 3675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:46.130877 3675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:46.136158 3675 catalog_manager.cc:1347] Generated new cluster ID: 38436dc26fdc4761979eb5e6437dbdb9
I20260810 22:56:46.136214 3675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:46.166672 3675 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:46.167969 3675 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:46.185036 3675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 505f973bc0bd4a11ad89ffe96ed01a61: Generated new TSK 0
I20260810 22:56:46.185787 3675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:56:46.209978 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:46.204854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.29.78.190:35335)
0810 22:56:46.204967 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:46.204972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:46.205169 (+ 197us) server_negotiation.cc:407] Connection header received
0810 22:56:46.205310 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:46.205313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:46.205384 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:46.205500 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:56:46.205712 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.206567 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.207428 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.207764 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.207859 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:46.208119 (+ 260us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:56:46.209591 (+ 1472us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 22:56:46.209802 (+ 211us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 22:56:46.210054 3668 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33866: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 22:56:46.211484 3697 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:46.204770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35335 (local address 127.0.0.1:33866)
0810 22:56:46.205060 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:46.205072 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:46.205185 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:46.205505 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:46.205507 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:46.205513 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:56:46.205612 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:46.205616 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.206708 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.206710 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:46.207311 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:46.207318 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.207876 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.207879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:46.207936 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:46.207958 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:56:46.209738 (+ 1780us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 22:56:46.211323 (+ 1585us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:35335: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36827/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1401,"thread_start_us":118,"threads_started":1}
I20260810 22:56:46.212405 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3592
E20260810 22:56:46.222922 3698 webserver.cc:533] Webserver: sslize failed:
E20260810 22:56:46.222985 3698 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T22:56:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T22:56:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:46Z Disabled control of system clock
I20260810 22:56:46.265043 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33199
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:44591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:33199
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/testchainca.cert with env {}
W20260810 22:56:46.370184 3703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:46.370409 3703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:46.370445 3703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:46.374245 3703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:46.374295 3703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:46.374312 3703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:46.374329 3703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:46.374344 3703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 22:56:46.374403 3703 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 22:56:46.374428 3703 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 22:56:46.379365 3703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:33199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:33199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:46.380318 3703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:46.381265 3703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:46.389251 3709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:46.389266 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:46.389266 3711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:46.389603 3703 server_base.cc:1034] running on GCE node
I20260810 22:56:46.390214 3703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:46.391244 3703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:46.392438 3703 hybrid_clock.cc:648] HybridClock initialized: now 1786402606392400 us; error 54 us; skew 500 ppm
I20260810 22:56:46.394371 3703 webserver.cc:460] Webserver started at http://127.29.78.190:40685/ using document root <none> and password file <none>
I20260810 22:56:46.394943 3703 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:46.395041 3703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:46.395293 3703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:46.397441 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "eb6f0241885c4a71a383257b6da94c86"
format_stamp: "Formatted at 2026-08-10 22:56:46 on dist-test-slave-6nmv"
server_key: "aea5d54ce4adc64b3230c1067a8eb4a1"
server_key_iv: "081bd2ea880dc55dddbed2868e347678"
server_key_version: "encryptionkey@0"
I20260810 22:56:46.398125 3703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "eb6f0241885c4a71a383257b6da94c86"
format_stamp: "Formatted at 2026-08-10 22:56:46 on dist-test-slave-6nmv"
server_key: "aea5d54ce4adc64b3230c1067a8eb4a1"
server_key_iv: "081bd2ea880dc55dddbed2868e347678"
server_key_version: "encryptionkey@0"
I20260810 22:56:46.401760 3703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:56:46.403973 3717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:46.405033 3703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:56:46.405210 3703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "eb6f0241885c4a71a383257b6da94c86"
format_stamp: "Formatted at 2026-08-10 22:56:46 on dist-test-slave-6nmv"
server_key: "aea5d54ce4adc64b3230c1067a8eb4a1"
server_key_iv: "081bd2ea880dc55dddbed2868e347678"
server_key_version: "encryptionkey@0"
I20260810 22:56:46.405330 3703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:46.428835 3703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:46.429510 3703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:46.429701 3703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:46.436843 3703 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:33199
I20260810 22:56:46.436851 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:33199 every 8 connection(s)
I20260810 22:56:46.437680 3703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:46.440227 3779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:46.440536 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3703
I20260810 22:56:46.440663 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:46.440994 30010 external_mini_cluster.cc:1428] Setting key 848fff66ce87ec61181aeb2c50a49e8b
I20260810 22:56:46.446014 3779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86: Bootstrap starting.
I20260810 22:56:46.448400 3779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:46.449108 3779 log.cc:824] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:46.449379 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:46.442265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.29.78.190:33199)
0810 22:56:46.442844 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:46.442866 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:46.442899 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:56:46.443485 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:46.443498 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:46.443845 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:46.444123 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:46.444426 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.445322 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.446505 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.446944 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.447051 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:46.447832 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:46.447836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:46.447846 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:56:46.448186 (+ 340us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:46.448233 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:56:46.448250 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:46.448316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:46.448498 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 22:56:46.448702 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
I20260810 22:56:46.450793 3779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86: No bootstrap required, opened a new log
I20260810 22:56:46.451879 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:56:46.454361 3779 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f0241885c4a71a383257b6da94c86" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33199 } }
I20260810 22:56:46.454574 3779 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:46.454615 3779 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6f0241885c4a71a383257b6da94c86, State: Initialized, Role: FOLLOWER
I20260810 22:56:46.454978 3779 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f0241885c4a71a383257b6da94c86" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33199 } }
I20260810 22:56:46.455101 3779 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:46.455148 3779 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:46.455235 3779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:46.456005 3779 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f0241885c4a71a383257b6da94c86" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33199 } }
I20260810 22:56:46.456272 3779 leader_election.cc:302] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb6f0241885c4a71a383257b6da94c86; no voters:
I20260810 22:56:46.456511 3779 leader_election.cc:288] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:46.456683 3784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:46.456944 3784 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [term 1 LEADER]: Becoming Leader. State: Replica: eb6f0241885c4a71a383257b6da94c86, State: Running, Role: LEADER
I20260810 22:56:46.457214 3784 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f0241885c4a71a383257b6da94c86" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33199 } }
I20260810 22:56:46.457628 3779 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:46.458968 3786 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb6f0241885c4a71a383257b6da94c86. Latest consensus state: current_term: 1 leader_uuid: "eb6f0241885c4a71a383257b6da94c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f0241885c4a71a383257b6da94c86" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33199 } } }
I20260810 22:56:46.459105 3786 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:46.459319 3785 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb6f0241885c4a71a383257b6da94c86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb6f0241885c4a71a383257b6da94c86" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 33199 } } }
I20260810 22:56:46.459401 3785 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:46.459564 3794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:46.462460 3794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:46.467804 3794 catalog_manager.cc:1347] Generated new cluster ID: b9ca01f098394f87aafcc093d8c59ab7
I20260810 22:56:46.467878 3794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:46.480804 3794 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:46.481614 3794 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:46.485877 3794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb6f0241885c4a71a383257b6da94c86: Generated new TSK 0
I20260810 22:56:46.486514 3794 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:56:46.962314 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:46.492471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.29.78.190:33199)
0810 22:56:46.492591 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:46.492598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:46.492774 (+ 176us) server_negotiation.cc:407] Connection header received
0810 22:56:46.492986 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:46.492988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:46.493031 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:46.493132 (+ 101us) server_negotiation.cc:226] Negotiated authn=JWT
0810 22:56:46.493354 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.493869 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.494691 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.494900 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.495004 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:46.495336 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 22:56:46.961759 (+466423us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 22:56:46.962154 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 22:56:46.962445 3782 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 22:56:46.962469 3810 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:46.492410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:33199 (local address 127.0.0.1:55884)
0810 22:56:46.492680 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:46.492692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:46.492820 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:46.493134 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:46.493137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:46.493144 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 22:56:46.493253 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:46.493264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.494011 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.494015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:46.494574 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:46.494582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:46.495025 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:46.495029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:46.495121 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:46.495141 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 22:56:46.962059 (+466918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 22:56:46.962256 (+ 197us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.29.78.190:33199: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
I20260810 22:56:46.963111 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3703
2026-08-10T22:56:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (729 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (121 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:47 dist-test-slave-6nmv krb5kdc[3823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:47 dist-test-slave-6nmv krb5kdc[3823](info): set up 2 sockets
Aug 10 22:56:47 dist-test-slave-6nmv krb5kdc[3823](info): commencing operation
krb5kdc: starting...
W20260810 22:56:48.256759 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:48 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:56:48.388614 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35287
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:48.495651 3838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:48.495921 3838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:48.495963 3838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:48.499421 3838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:48.499471 3838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:48.499488 3838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:48.499506 3838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:48.499519 3838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:48.504213 3838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:48.505240 3838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:48.506337 3838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:48.514529 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:48.514585 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:48.514479 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:48.514747 3838 server_base.cc:1034] running on GCE node
I20260810 22:56:48.515336 3838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:48.516499 3838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:48.517691 3838 hybrid_clock.cc:648] HybridClock initialized: now 1786402608517660 us; error 45 us; skew 500 ppm
Aug 10 22:56:48 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402608, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:48.520471 3838 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:56:48.521322 3838 webserver.cc:460] Webserver started at http://127.29.78.190:39741/ using document root <none> and password file <none>
I20260810 22:56:48.521930 3838 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:48.522010 3838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:48.522274 3838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:48.524334 3838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "af2d3a13e79a4b21a91e3729d423dc9f"
format_stamp: "Formatted at 2026-08-10 22:56:48 on dist-test-slave-6nmv"
server_key: "f67876e6da9915f1ab548b121f67583c"
server_key_iv: "7f9a9bb6da94655d2d3f3b9feab40b5f"
server_key_version: "encryptionkey@0"
I20260810 22:56:48.524968 3838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "af2d3a13e79a4b21a91e3729d423dc9f"
format_stamp: "Formatted at 2026-08-10 22:56:48 on dist-test-slave-6nmv"
server_key: "f67876e6da9915f1ab548b121f67583c"
server_key_iv: "7f9a9bb6da94655d2d3f3b9feab40b5f"
server_key_version: "encryptionkey@0"
I20260810 22:56:48.528712 3838 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:56:48.531083 3858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:48.532246 3838 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:48.532374 3838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "af2d3a13e79a4b21a91e3729d423dc9f"
format_stamp: "Formatted at 2026-08-10 22:56:48 on dist-test-slave-6nmv"
server_key: "f67876e6da9915f1ab548b121f67583c"
server_key_iv: "7f9a9bb6da94655d2d3f3b9feab40b5f"
server_key_version: "encryptionkey@0"
I20260810 22:56:48.532473 3838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:48.549293 3838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:48.553006 3838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:48.553249 3838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:48.564028 3838 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35287
I20260810 22:56:48.564328 3915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35287 every 8 connection(s)
I20260810 22:56:48.565218 3838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:48.568945 3917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:48.575490 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3838
I20260810 22:56:48.575635 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:48.575961 30010 external_mini_cluster.cc:1428] Setting key dc525cccf0b33fdb817ea138354d7216
I20260810 22:56:48.576040 3917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f: Bootstrap starting.
I20260810 22:56:48.578889 3917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:48.579664 3917 log.cc:824] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:48.582048 3917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f: No bootstrap required, opened a new log
Aug 10 22:56:48 dist-test-slave-6nmv krb5kdc[3823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:48.586444 3917 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35287 } }
I20260810 22:56:48.586755 3917 raft_consensus.cc:374] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:48.586822 3917 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2d3a13e79a4b21a91e3729d423dc9f, State: Initialized, Role: FOLLOWER
I20260810 22:56:48.587352 3917 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35287 } }
I20260810 22:56:48.587525 3917 raft_consensus.cc:388] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:48.587594 3917 raft_consensus.cc:482] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:48.587699 3917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:48.589007 3917 raft_consensus.cc:504] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35287 } }
I20260810 22:56:48.589375 3917 leader_election.cc:302] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af2d3a13e79a4b21a91e3729d423dc9f; no voters:
I20260810 22:56:48.589777 3917 leader_election.cc:288] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:48.589896 3922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:48.590164 3922 raft_consensus.cc:686] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [term 1 LEADER]: Becoming Leader. State: Replica: af2d3a13e79a4b21a91e3729d423dc9f, State: Running, Role: LEADER
I20260810 22:56:48.590454 3922 consensus_queue.cc:237] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35287 } }
I20260810 22:56:48.591188 3917 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:48.591500 3922 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader af2d3a13e79a4b21a91e3729d423dc9f. Latest consensus state: current_term: 1 leader_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35287 } } }
I20260810 22:56:48.591624 3922 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:48.592120 3923 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2d3a13e79a4b21a91e3729d423dc9f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35287 } } }
I20260810 22:56:48.592280 3923 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:48.593181 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:48.577416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47946 (local address 127.29.78.190:35287)
0810 22:56:48.578007 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:48.578037 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:48.578094 (+ 57us) server_negotiation.cc:407] Connection header received
0810 22:56:48.578837 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:48.578854 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:48.579245 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:48.579563 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:48.580465 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:48.581919 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:48.583187 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:48.583844 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:48.584006 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:48.587166 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:48.587172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:48.587187 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:48.587217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:48.589253 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:48.589801 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:48.589830 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:48.589836 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:48.590093 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:48.590492 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:48.590495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:48.590497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:48.591069 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:48.591240 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:48.591765 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 22:56:48.592098 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
W20260810 22:56:48.596119 3939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:56:48.596199 3939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:56:48.597955 3926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:48.598713 3926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:48.605098 3926 catalog_manager.cc:1347] Generated new cluster ID: f33a0ba17eb64f3096dfb07edce3df97
I20260810 22:56:48.605161 3926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:48.612612 3926 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:48.613794 3926 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:48.620124 3926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af2d3a13e79a4b21a91e3729d423dc9f: Generated new TSK 0
I20260810 22:56:48.620826 3926 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:56:48.656845 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35287
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:48.764199 3945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:48.764432 3945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:48.764470 3945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:48.768324 3945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:48.768373 3945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:48.768453 3945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:48.773247 3945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:48.774252 3945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:48.775213 3945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:48.784091 3952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:48.784116 3955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:48.784116 3953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:48.784240 3945 server_base.cc:1034] running on GCE node
I20260810 22:56:48.784648 3945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:48.785236 3945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:48.786407 3945 hybrid_clock.cc:648] HybridClock initialized: now 1786402608786403 us; error 36 us; skew 500 ppm
Aug 10 22:56:48 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402608, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:48.788931 3945 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:56:48.789710 3945 webserver.cc:460] Webserver started at http://127.29.78.129:40849/ using document root <none> and password file <none>
I20260810 22:56:48.790230 3945 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:48.790308 3945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:48.790527 3945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:48.792650 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "0e41e045645d47ab9c7df402e109fa61"
format_stamp: "Formatted at 2026-08-10 22:56:48 on dist-test-slave-6nmv"
server_key: "67b35cfccf5aed2e76856e0ad3a85836"
server_key_iv: "433adadd0d71b2eaace80a26eb069350"
server_key_version: "encryptionkey@0"
I20260810 22:56:48.793213 3945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "0e41e045645d47ab9c7df402e109fa61"
format_stamp: "Formatted at 2026-08-10 22:56:48 on dist-test-slave-6nmv"
server_key: "67b35cfccf5aed2e76856e0ad3a85836"
server_key_iv: "433adadd0d71b2eaace80a26eb069350"
server_key_version: "encryptionkey@0"
I20260810 22:56:48.796896 3945 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:56:48.799095 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:48.800177 3945 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:48.800307 3945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "0e41e045645d47ab9c7df402e109fa61"
format_stamp: "Formatted at 2026-08-10 22:56:48 on dist-test-slave-6nmv"
server_key: "67b35cfccf5aed2e76856e0ad3a85836"
server_key_iv: "433adadd0d71b2eaace80a26eb069350"
server_key_version: "encryptionkey@0"
I20260810 22:56:48.800417 3945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:48.821295 3945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:48.824043 3945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:48.824259 3945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:48.824826 3945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:48.825624 3945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:48.825693 3945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:48.825760 3945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:48.825806 3945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:48.834630 3945 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:43257
I20260810 22:56:48.834658 4085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:43257 every 8 connection(s)
I20260810 22:56:48.835443 3945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:48.843246 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 3945
I20260810 22:56:48.843376 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:48.843653 30010 external_mini_cluster.cc:1428] Setting key 4d9976d6e570c7045caf4420f982721c
Aug 10 22:56:48 dist-test-slave-6nmv krb5kdc[3823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402608, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:48.848694 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:48.837251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:57163 (local address 127.29.78.190:35287)
0810 22:56:48.837389 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:48.837392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:48.838147 (+ 755us) server_negotiation.cc:407] Connection header received
0810 22:56:48.839484 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:48.839487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:48.839544 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:48.839637 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:48.840695 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:48.841202 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:48.842433 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:48.842671 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:48.842775 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:48.845166 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:48.845169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:48.845171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:48.845195 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:48.846845 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:48.847286 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:48.847292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:48.847295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:48.847394 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:48.847688 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:48.847692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:48.847693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:48.847903 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:48.848001 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:48.848468 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 22:56:48.848582 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:56:48.849740 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:48.837565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35287 (local address 127.29.78.129:57163)
0810 22:56:48.838013 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:48.838054 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:48.839280 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:48.839799 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:48.839805 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:48.840121 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:48.840558 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:48.840570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:48.841319 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:48.841322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:48.842302 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:48.842312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:48.842811 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:48.842815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:48.842971 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:48.843648 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:48.843662 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:48.845027 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:48.846955 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:48.846959 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:48.846968 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:48.847197 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:48.847485 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:48.847487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:48.847488 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:48.847606 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:48.847990 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:48.847994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:48.848202 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 22:56:48.848416 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 22:56:48.848734 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":106,"threads_started":1}
I20260810 22:56:48.850925 4086 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35287
I20260810 22:56:48.851164 4086 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:48.851665 4086 heartbeater.cc:511] Master 127.29.78.190:35287 requested a full tablet report, sending...
I20260810 22:56:48.853204 3867 ts_manager.cc:194] Registered new tserver with Master: 0e41e045645d47ab9c7df402e109fa61 (127.29.78.129:43257)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:48.854398 3867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:57163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:56:48.872673 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35287
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:48.983798 4093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:48.984056 4093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:48.984097 4093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:48.987965 4093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:48.988018 4093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:48.988124 4093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:48.993263 4093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:48.994295 4093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:48.995358 4093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:49.004583 4098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:49.004734 4093 server_base.cc:1034] running on GCE node
W20260810 22:56:49.007061 4099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:49.007140 4101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:49.007617 4093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:49.008342 4093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:49.009544 4093 hybrid_clock.cc:648] HybridClock initialized: now 1786402609009530 us; error 40 us; skew 500 ppm
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.012526 4093 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:56:49.013518 4093 webserver.cc:460] Webserver started at http://127.29.78.130:33103/ using document root <none> and password file <none>
I20260810 22:56:49.014199 4093 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:49.014295 4093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:49.014577 4093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:49.016733 4093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "9dde838fce9c409495100c0075b29fa8"
format_stamp: "Formatted at 2026-08-10 22:56:49 on dist-test-slave-6nmv"
server_key: "d4978d1d20337be97233e95b179679a3"
server_key_iv: "d3bed9341df0b8c9a9bac7f85ed27688"
server_key_version: "encryptionkey@0"
I20260810 22:56:49.017356 4093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "9dde838fce9c409495100c0075b29fa8"
format_stamp: "Formatted at 2026-08-10 22:56:49 on dist-test-slave-6nmv"
server_key: "d4978d1d20337be97233e95b179679a3"
server_key_iv: "d3bed9341df0b8c9a9bac7f85ed27688"
server_key_version: "encryptionkey@0"
I20260810 22:56:49.021487 4093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 22:56:49.024150 4109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:49.025280 4093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:49.025449 4093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "9dde838fce9c409495100c0075b29fa8"
format_stamp: "Formatted at 2026-08-10 22:56:49 on dist-test-slave-6nmv"
server_key: "d4978d1d20337be97233e95b179679a3"
server_key_iv: "d3bed9341df0b8c9a9bac7f85ed27688"
server_key_version: "encryptionkey@0"
I20260810 22:56:49.025560 4093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:49.044899 4093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:49.047705 4093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:49.047896 4093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:49.048429 4093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:49.049279 4093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:49.049376 4093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:49.049463 4093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:49.049508 4093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:49.058353 4093 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:33563
I20260810 22:56:49.058385 4235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:33563 every 8 connection(s)
I20260810 22:56:49.059222 4093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:49.069170 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 4093
I20260810 22:56:49.069300 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:49.069577 30010 external_mini_cluster.cc:1428] Setting key febda7370a1951c35819c3713dbc5389
I20260810 22:56:49.072158 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:49.060896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:58657 (local address 127.29.78.190:35287)
0810 22:56:49.061023 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.061026 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.061731 (+ 705us) server_negotiation.cc:407] Connection header received
0810 22:56:49.063086 (+ 1355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.063089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.063143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.063234 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.064264 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.064721 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.065791 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.066070 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.066183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:49.068495 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:49.068498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:49.068500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:49.068522 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:49.069994 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.070568 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.070572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.070573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.070654 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.071032 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.071035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.071036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.071237 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:49.071316 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:49.071939 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 22:56:49.072042 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:56:49.073396 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:49.061203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35287 (local address 127.29.78.130:58657)
0810 22:56:49.061592 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:49.061632 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:49.062901 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:49.063397 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:49.063403 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:49.063714 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:49.064135 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:49.064146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.064831 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.064834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:49.065665 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:49.065675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.066221 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.066225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:49.066396 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:49.066931 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:49.066947 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:49.068318 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:49.070117 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:49.070121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:49.070131 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:49.070476 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:49.070747 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:49.070749 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:49.070750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:49.070935 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:49.071343 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:49.071348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:49.071687 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 22:56:49.071909 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 22:56:49.072273 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":98,"threads_started":1}
I20260810 22:56:49.074525 4236 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35287
I20260810 22:56:49.074755 4236 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:49.075284 4236 heartbeater.cc:511] Master 127.29.78.190:35287 requested a full tablet report, sending...
I20260810 22:56:49.076421 3867 ts_manager.cc:194] Registered new tserver with Master: 9dde838fce9c409495100c0075b29fa8 (127.29.78.130:33563)
I20260810 22:56:49.076987 3867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:58657
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:56:49.098449 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35287
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:41105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:49.201423 4241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:49.201699 4241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:49.201771 4241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:49.205773 4241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:49.205866 4241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:49.205977 4241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:49.210731 4241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:41105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:49.211650 4241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:49.212632 4241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:49.221735 4249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:49.221782 4251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:49.221782 4248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:49.221951 4241 server_base.cc:1034] running on GCE node
I20260810 22:56:49.222566 4241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:49.223172 4241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:49.224368 4241 hybrid_clock.cc:648] HybridClock initialized: now 1786402609224355 us; error 29 us; skew 500 ppm
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.227177 4241 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:56:49.228025 4241 webserver.cc:460] Webserver started at http://127.29.78.131:39557/ using document root <none> and password file <none>
I20260810 22:56:49.228592 4241 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:49.228650 4241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:49.228838 4241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:49.230774 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "e457d917e37140c295dfff68f48ff78e"
format_stamp: "Formatted at 2026-08-10 22:56:49 on dist-test-slave-6nmv"
server_key: "490f283a052d5f9c5f65caa39d3f2465"
server_key_iv: "642b9a17383f6f0ecc8d6442a7c84c85"
server_key_version: "encryptionkey@0"
I20260810 22:56:49.231271 4241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "e457d917e37140c295dfff68f48ff78e"
format_stamp: "Formatted at 2026-08-10 22:56:49 on dist-test-slave-6nmv"
server_key: "490f283a052d5f9c5f65caa39d3f2465"
server_key_iv: "642b9a17383f6f0ecc8d6442a7c84c85"
server_key_version: "encryptionkey@0"
I20260810 22:56:49.234716 4241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:56:49.237056 4258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:49.238155 4241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:49.238260 4241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "e457d917e37140c295dfff68f48ff78e"
format_stamp: "Formatted at 2026-08-10 22:56:49 on dist-test-slave-6nmv"
server_key: "490f283a052d5f9c5f65caa39d3f2465"
server_key_iv: "642b9a17383f6f0ecc8d6442a7c84c85"
server_key_version: "encryptionkey@0"
I20260810 22:56:49.238349 4241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:49.255208 4241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:49.258342 4241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:49.258558 4241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:49.259146 4241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:49.259891 4241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:49.259933 4241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:49.259992 4241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:49.260032 4241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:49.268537 4241 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:36395
I20260810 22:56:49.268559 4383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:36395 every 8 connection(s)
I20260810 22:56:49.269352 4241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:49.274092 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 4241
I20260810 22:56:49.274186 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:49.274452 30010 external_mini_cluster.cc:1428] Setting key 632502102f0775b6754fe089b7150e4f
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:56:49.282548 3920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:49.271129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:41619 (local address 127.29.78.190:35287)
0810 22:56:49.271265 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.271269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.272051 (+ 782us) server_negotiation.cc:407] Connection header received
0810 22:56:49.273343 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.273346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.273391 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.273462 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.274623 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.275230 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.276543 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.276771 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.276870 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:49.279254 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:49.279258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:49.279260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:49.279283 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:49.280566 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.281075 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.281083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.281084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.281157 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.281480 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.281485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.281486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.281672 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:49.281775 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:49.282264 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 22:56:49.282387 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:56:49.283243 4386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:49.271450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35287 (local address 127.29.78.131:41619)
0810 22:56:49.271907 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:49.271946 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:49.273167 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:49.273613 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:49.273618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:49.273983 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:49.274466 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:49.274494 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.275347 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.275350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:49.276429 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:49.276438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.276909 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.276913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:49.277056 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:49.277496 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:49.277511 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:49.279075 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:49.280690 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:49.280696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:49.280708 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:49.280971 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:49.281260 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:49.281263 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:49.281265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:49.281396 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:49.281774 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:49.281780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:49.282045 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 22:56:49.282223 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 22:56:49.282516 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
I20260810 22:56:49.284155 4384 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35287
I20260810 22:56:49.284328 4384 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:49.284729 4384 heartbeater.cc:511] Master 127.29.78.190:35287 requested a full tablet report, sending...
I20260810 22:56:49.285585 3867 ts_manager.cc:194] Registered new tserver with Master: e457d917e37140c295dfff68f48ff78e (127.29.78.131:36395)
I20260810 22:56:49.286183 3867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:41619
I20260810 22:56:49.287882 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/test-admin.keytab.
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.303632 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 22:56:49.306193 4392 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:49.304881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.29.78.129:43257)
0810 22:56:49.305172 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.305184 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.305205 (+ 21us) server_negotiation.cc:407] Connection header received
0810 22:56:49.305246 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.305250 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.305402 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.305543 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.306021 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52918: BlockingRecv error: recv got EOF from 127.0.0.1:52918 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":121,"threads_started":1}
W20260810 22:56:49.306238 30010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5cc' of type FILE
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.307293 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.308151 30010 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:49.316910 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:49.308885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.29.78.129:43257)
0810 22:56:49.308997 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.309001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.309075 (+ 74us) server_negotiation.cc:407] Connection header received
0810 22:56:49.309217 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.309219 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.309251 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.309318 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.309770 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.310560 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.311286 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.311665 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.311794 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:49.313790 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:49.313798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:49.313843 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:49.313867 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:49.315198 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.315626 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.315632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.315634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.315719 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.315995 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.315999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.316001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.316233 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:49.316437 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:49.316648 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 22:56:49.316790 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 22:56:49.320173 4392 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:49.319252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52938 (local address 127.29.78.129:43257)
0810 22:56:49.319351 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.319355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.319439 (+ 84us) server_negotiation.cc:407] Connection header received
0810 22:56:49.319562 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.319565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.319599 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.319666 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.320086 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52938: BlockingRecv error: recv got EOF from 127.0.0.1:52938 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 22:56:49.320293 30010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5cc' of type FILE
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.321147 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.321977 30010 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:49.329288 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:49.322608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.29.78.129:43257)
0810 22:56:49.322718 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.322720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.322754 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:56:49.322830 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.322832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.322860 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.322918 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.323312 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.323760 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.324404 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.324591 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.324656 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:49.326748 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:49.326750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:49.326752 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:49.326770 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:49.327824 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.328187 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.328189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.328191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.328253 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.328463 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.328465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.328467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.328627 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:49.328712 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:49.328949 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 22:56:49.329106 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 22:56:49.331913 4392 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:49.330954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52960 (local address 127.29.78.129:43257)
0810 22:56:49.331086 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.331089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.331184 (+ 95us) server_negotiation.cc:407] Connection header received
0810 22:56:49.331288 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.331291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.331327 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.331398 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.331796 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52960: BlockingRecv error: recv got EOF from 127.0.0.1:52960 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 22:56:49.332074 30010 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402578782454-30010-0/krb5kdc/krb5cc' of type FILE
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.333007 30010 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:49.333885 30010 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[3823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:56:49.341548 4392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:49.334561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52976 (local address 127.29.78.129:43257)
0810 22:56:49.334655 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:49.334658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:49.334773 (+ 115us) server_negotiation.cc:407] Connection header received
0810 22:56:49.334850 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:49.334852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:49.334883 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:49.334942 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:49.335334 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.335829 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.336698 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:49.336947 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:49.337031 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:49.339022 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:49.339025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:49.339028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:49.339053 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:49.340121 (+ 1068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.340488 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.340492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.340493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.340567 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:49.340823 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:49.340826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:49.340828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:49.340983 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:49.341101 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:49.341319 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 22:56:49.341429 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 22:56:49.342476 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3945
I20260810 22:56:49.351789 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 4093
I20260810 22:56:49.358434 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 4241
I20260810 22:56:49.365741 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 3838
2026-08-10T22:56:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2160 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[4408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[4408](info): set up 2 sockets
Aug 10 22:56:49 dist-test-slave-6nmv krb5kdc[4408](info): commencing operation
krb5kdc: starting...
W20260810 22:56:50.402204 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:50 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
I20260810 22:56:50.522410 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42665
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:50.626761 4422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:50.627032 4422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:50.627117 4422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:50.630921 4422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:50.631024 4422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:50.631076 4422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:50.631141 4422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:50.631181 4422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:50.636360 4422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:42665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:50.637387 4422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:50.638384 4422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:50.646113 4431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:50.646260 4422 server_base.cc:1034] running on GCE node
W20260810 22:56:50.646248 4429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:50.646124 4428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:50.646993 4422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:50.648028 4422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:50.649219 4422 hybrid_clock.cc:648] HybridClock initialized: now 1786402610649199 us; error 29 us; skew 500 ppm
Aug 10 22:56:50 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:50.651811 4422 init.cc:377] Logged in from keytab as oryx/127.29.78.190@KRBTEST.COM (short username oryx)
I20260810 22:56:50.652678 4422 webserver.cc:460] Webserver started at http://127.29.78.190:45399/ using document root <none> and password file <none>
I20260810 22:56:50.653185 4422 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:50.653273 4422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:50.653492 4422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:50.655375 4422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "6318f97f3e634a06ab2be1f8cddbcecb"
format_stamp: "Formatted at 2026-08-10 22:56:50 on dist-test-slave-6nmv"
server_key: "9f149a9a5f48e8090c45dc4f056efd6f"
server_key_iv: "66f71693f7be81ad8aef56b8d0b97fd5"
server_key_version: "encryptionkey@0"
I20260810 22:56:50.655995 4422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "6318f97f3e634a06ab2be1f8cddbcecb"
format_stamp: "Formatted at 2026-08-10 22:56:50 on dist-test-slave-6nmv"
server_key: "9f149a9a5f48e8090c45dc4f056efd6f"
server_key_iv: "66f71693f7be81ad8aef56b8d0b97fd5"
server_key_version: "encryptionkey@0"
I20260810 22:56:50.659593 4422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:56:50.661859 4439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:50.662971 4422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:50.663105 4422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "6318f97f3e634a06ab2be1f8cddbcecb"
format_stamp: "Formatted at 2026-08-10 22:56:50 on dist-test-slave-6nmv"
server_key: "9f149a9a5f48e8090c45dc4f056efd6f"
server_key_iv: "66f71693f7be81ad8aef56b8d0b97fd5"
server_key_version: "encryptionkey@0"
I20260810 22:56:50.663210 4422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:50.675374 4422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:50.678279 4422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:50.678519 4422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:50.685578 4422 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:42665
I20260810 22:56:50.685938 4495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:42665 every 8 connection(s)
I20260810 22:56:50.686611 4422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:50.687901 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 4422
I20260810 22:56:50.688040 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:50.688359 30010 external_mini_cluster.cc:1428] Setting key b53eb0b07562c223266ff6652f44d745
I20260810 22:56:50.689779 4496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:50.695909 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: Bootstrap starting.
Aug 10 22:56:50 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:50.698326 4496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:50.698940 4496 log.cc:824] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:50.700842 4496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: No bootstrap required, opened a new log
I20260810 22:56:50.702137 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:50.689653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.29.78.190:42665)
0810 22:56:50.690062 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:50.690079 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:50.690116 (+ 37us) server_negotiation.cc:407] Connection header received
0810 22:56:50.690868 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:50.690887 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:50.691236 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:50.691558 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:50.692186 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:50.693565 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:50.694335 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:50.694964 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:50.695140 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:50.697787 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:50.697793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:50.697842 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:50.697878 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:50.699639 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:50.700057 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:50.700060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:50.700064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:50.700158 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:50.700408 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:50.700409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:50.700410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:50.700731 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:50.700876 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:50.701119 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 22:56:50.701392 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":160,"threads_started":1}
I20260810 22:56:50.704317 4496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } }
I20260810 22:56:50.704490 4496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:50.704563 4496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6318f97f3e634a06ab2be1f8cddbcecb, State: Initialized, Role: FOLLOWER
I20260810 22:56:50.704938 4496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } }
I20260810 22:56:50.705089 4496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:50.705152 4496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:50.705252 4496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:50.706085 4496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } }
I20260810 22:56:50.706421 4496 leader_election.cc:302] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6318f97f3e634a06ab2be1f8cddbcecb; no voters:
I20260810 22:56:50.706777 4496 leader_election.cc:288] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:50.706894 4502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:50.707139 4502 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 1 LEADER]: Becoming Leader. State: Replica: 6318f97f3e634a06ab2be1f8cddbcecb, State: Running, Role: LEADER
I20260810 22:56:50.707517 4502 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } }
I20260810 22:56:50.707947 4496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:50.709053 4503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6318f97f3e634a06ab2be1f8cddbcecb. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } }
I20260810 22:56:50.709187 4503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:50.709298 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } }
I20260810 22:56:50.709399 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:50.709578 4512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:50.712539 4512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:50.717490 4512 catalog_manager.cc:1347] Generated new cluster ID: 2a5f72c4338d4d55b84ee7c2c4923a66
I20260810 22:56:50.717555 4512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:50.730893 4512 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:50.731812 4512 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:50.737154 4512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: Generated new TSK 0
I20260810 22:56:50.737758 4512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
I20260810 22:56:50.770169 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42665
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:50.890007 4527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:50.890225 4527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:50.890262 4527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:50.894160 4527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:50.894203 4527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:50.894250 4527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:50.898916 4527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:50.899750 4527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:50.900671 4527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:50.910831 4534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:50.910849 4536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:50.910858 4533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:50.911100 4527 server_base.cc:1034] running on GCE node
I20260810 22:56:50.911624 4527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:50.912238 4527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:50.913419 4527 hybrid_clock.cc:648] HybridClock initialized: now 1786402610913401 us; error 29 us; skew 500 ppm
Aug 10 22:56:50 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:50.916270 4527 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260810 22:56:50.917219 4527 webserver.cc:460] Webserver started at http://127.29.78.129:42027/ using document root <none> and password file <none>
I20260810 22:56:50.917779 4527 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:50.917913 4527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:50.918159 4527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:50.920342 4527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "5fd6b2a2130d427c8bbe156ed85f7d06"
format_stamp: "Formatted at 2026-08-10 22:56:50 on dist-test-slave-6nmv"
server_key: "e0ae2fd19a315d0daff6bc80c197fac5"
server_key_iv: "ca023d3081a0ab0064b0b3bc83021eb7"
server_key_version: "encryptionkey@0"
I20260810 22:56:50.920949 4527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "5fd6b2a2130d427c8bbe156ed85f7d06"
format_stamp: "Formatted at 2026-08-10 22:56:50 on dist-test-slave-6nmv"
server_key: "e0ae2fd19a315d0daff6bc80c197fac5"
server_key_iv: "ca023d3081a0ab0064b0b3bc83021eb7"
server_key_version: "encryptionkey@0"
I20260810 22:56:50.925343 4527 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 22:56:50.927845 4543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:50.929031 4527 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:56:50.929162 4527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "5fd6b2a2130d427c8bbe156ed85f7d06"
format_stamp: "Formatted at 2026-08-10 22:56:50 on dist-test-slave-6nmv"
server_key: "e0ae2fd19a315d0daff6bc80c197fac5"
server_key_iv: "ca023d3081a0ab0064b0b3bc83021eb7"
server_key_version: "encryptionkey@0"
I20260810 22:56:50.929270 4527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:50.943454 4527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:50.946942 4527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:50.947130 4527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:50.947701 4527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:50.948508 4527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:50.948629 4527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:50.948740 4527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:50.948784 4527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:50.958426 4527 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:36833
I20260810 22:56:50.958446 4672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:36833 every 8 connection(s)
I20260810 22:56:50.959349 4527 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:50.967087 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 4527
I20260810 22:56:50.967247 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:50.967612 30010 external_mini_cluster.cc:1428] Setting key ca8405fbb01b772785dc96aaebbdd0ef
Aug 10 22:56:50 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:50.972711 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:50.961088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:36339 (local address 127.29.78.190:42665)
0810 22:56:50.961257 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:50.961263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:50.962061 (+ 798us) server_negotiation.cc:407] Connection header received
0810 22:56:50.963328 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:50.963332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:50.963399 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:50.963478 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:50.964525 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:50.965067 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:50.966214 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:50.966474 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:50.966587 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:50.969018 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:50.969021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:50.969023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:50.969048 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:50.970381 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:50.970936 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:50.970940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:50.970941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:50.971024 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:50.971423 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:50.971425 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:50.971426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:50.971617 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:50.971711 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:50.972333 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 22:56:50.972476 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:56:50.973272 4675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:50.961460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:36339)
0810 22:56:50.961931 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:50.961969 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:50.963130 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:50.963620 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:50.963626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:50.963963 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:50.964392 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:50.964405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:50.965171 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:50.965173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:50.966073 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:50.966081 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:50.966622 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:50.966626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:50.966815 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:50.967382 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:50.967397 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:50.968876 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:50.970502 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:50.970507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:50.970519 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:50.970829 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:50.971129 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:50.971133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:50.971136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:50.971312 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:50.971731 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:50.971738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:50.972025 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 22:56:50.972249 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 22:56:50.972484 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":126,"threads_started":1}
I20260810 22:56:50.974545 4673 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42665
I20260810 22:56:50.974776 4673 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:50.975203 4673 heartbeater.cc:511] Master 127.29.78.190:42665 requested a full tablet report, sending...
I20260810 22:56:50.976934 4458 ts_manager.cc:194] Registered new tserver with Master: 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129:36833)
WARNING: no policy specified for oryx/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:56:50.978605 4458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:36339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
I20260810 22:56:50.994777 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42665
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:51.099670 4679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:51.099910 4679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:51.099973 4679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:51.103366 4679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:51.103435 4679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:51.103518 4679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:51.108461 4679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:51.109455 4679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:51.110427 4679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:56:51.119436 4679 server_base.cc:1034] running on GCE node
W20260810 22:56:51.119431 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.119468 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.119572 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:51.119923 4679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:51.120517 4679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:51.121706 4679 hybrid_clock.cc:648] HybridClock initialized: now 1786402611121695 us; error 26 us; skew 500 ppm
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:51.124430 4679 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260810 22:56:51.125226 4679 webserver.cc:460] Webserver started at http://127.29.78.130:43137/ using document root <none> and password file <none>
I20260810 22:56:51.125744 4679 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:51.125859 4679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:51.126089 4679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:51.128048 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "38f13c2dd9314170b397a16a4a650007"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "8c8bac140dc118d42100bbaa321e9c4d"
server_key_iv: "51a40ac54cf98a3ba463924ee48891f1"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.128582 4679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "38f13c2dd9314170b397a16a4a650007"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "8c8bac140dc118d42100bbaa321e9c4d"
server_key_iv: "51a40ac54cf98a3ba463924ee48891f1"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.131924 4679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 22:56:51.134132 4695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.135169 4679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:56:51.135295 4679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "38f13c2dd9314170b397a16a4a650007"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "8c8bac140dc118d42100bbaa321e9c4d"
server_key_iv: "51a40ac54cf98a3ba463924ee48891f1"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.135409 4679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:51.153069 4679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:51.156211 4679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:51.156409 4679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:51.156983 4679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:51.157737 4679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:51.157799 4679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.157878 4679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:51.157920 4679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.166790 4679 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:42085
I20260810 22:56:51.166864 4821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:42085 every 8 connection(s)
I20260810 22:56:51.167658 4679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:51.170493 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 4679
I20260810 22:56:51.170607 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:51.170993 30010 external_mini_cluster.cc:1428] Setting key a6a1863e27eb32fe0b2a91801834b667
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:56:51.182178 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.169433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:58021 (local address 127.29.78.190:42665)
0810 22:56:51.169572 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.169575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.170329 (+ 754us) server_negotiation.cc:407] Connection header received
0810 22:56:51.172045 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.172048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.172094 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.172176 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.173751 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.174325 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.175532 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.175886 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.175970 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.178711 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.178714 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.178716 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.178737 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.180081 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.180654 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.180658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.180660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.180752 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.181089 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.181094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.181097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.181348 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.181434 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.181922 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.182069 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:56:51.182965 4824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.169690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:58021)
0810 22:56:51.170157 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.170198 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.171846 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.172390 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.172399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.172933 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.173596 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.173628 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.174442 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.174448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.175402 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.175411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.176035 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.176041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.176272 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.176906 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.176929 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.178558 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.180206 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.180211 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.180222 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.180537 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.180846 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.180848 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.180849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.180973 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.181438 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.181442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.181649 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.181851 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.182155 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":162,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.131@KRBTEST.COM" created.
I20260810 22:56:51.184175 4822 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42665
I20260810 22:56:51.184392 4822 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:51.184813 4822 heartbeater.cc:511] Master 127.29.78.190:42665 requested a full tablet report, sending...
I20260810 22:56:51.185956 4458 ts_manager.cc:194] Registered new tserver with Master: 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085)
I20260810 22:56:51.186549 4458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:58021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
I20260810 22:56:51.199019 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42665
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:51.303761 4827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:51.304000 4827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:51.304040 4827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:51.307996 4827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:51.308043 4827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:51.308130 4827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:51.313031 4827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:42665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:51.313957 4827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:51.314842 4827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:56:51.323997 4827 server_base.cc:1034] running on GCE node
W20260810 22:56:51.323956 4832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.324012 4834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.323956 4836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:51.324532 4827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:51.325114 4827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:51.326287 4827 hybrid_clock.cc:648] HybridClock initialized: now 1786402611326268 us; error 34 us; skew 500 ppm
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:51.328876 4827 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260810 22:56:51.329618 4827 webserver.cc:460] Webserver started at http://127.29.78.131:45575/ using document root <none> and password file <none>
I20260810 22:56:51.330178 4827 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:51.330266 4827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:51.330477 4827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:51.332405 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "0465065cac1c434a9cec3e07c19a8590"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "6eb18e9168f76b1f3d27e616e09cc2f5"
server_key_iv: "ceca3766dfe81ecc9fa00bad471eb161"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.333045 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "0465065cac1c434a9cec3e07c19a8590"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "6eb18e9168f76b1f3d27e616e09cc2f5"
server_key_iv: "ceca3766dfe81ecc9fa00bad471eb161"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.336849 4827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:56:51.339141 4844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.340277 4827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:51.340389 4827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "0465065cac1c434a9cec3e07c19a8590"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "6eb18e9168f76b1f3d27e616e09cc2f5"
server_key_iv: "ceca3766dfe81ecc9fa00bad471eb161"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.340479 4827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:51.361552 4827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:51.364137 4827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:51.364313 4827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:51.364908 4827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:51.365643 4827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:51.365707 4827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.365773 4827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:51.365830 4827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.374085 4827 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:37281
I20260810 22:56:51.374115 4973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:37281 every 8 connection(s)
I20260810 22:56:51.374905 4827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:51.375078 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 4827
I20260810 22:56:51.375197 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:51.375560 30010 external_mini_cluster.cc:1428] Setting key 449ba4bb42dd4135170dcc3ccab6e8df
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:51.387549 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.377989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:37001 (local address 127.29.78.190:42665)
0810 22:56:51.378143 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.378149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.378165 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:56:51.379125 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.379128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.379181 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.379271 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.380357 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.380879 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.381983 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.382192 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.382290 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.384567 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.384570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.384572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.384597 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.385875 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.386328 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.386330 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.386332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.386409 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.386681 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.386683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.386684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.386887 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.386957 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.387347 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.387438 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:56:51.388363 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.377078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:37001)
0810 22:56:51.377551 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.377615 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.378944 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.379447 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.379453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.379789 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.380224 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.380237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.380992 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.380995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.381870 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.381885 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.382327 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.382330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.382534 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.383040 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.383054 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.384430 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.385982 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.385986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.385996 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.386241 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.386481 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.386483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.386484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.386607 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.386962 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.386966 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.387171 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.387308 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.387532 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":123,"threads_started":1}
I20260810 22:56:51.389479 4974 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42665
I20260810 22:56:51.389703 4974 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:51.390156 4974 heartbeater.cc:511] Master 127.29.78.190:42665 requested a full tablet report, sending...
I20260810 22:56:51.391003 4458 ts_manager.cc:194] Registered new tserver with Master: 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131:37281)
I20260810 22:56:51.391515 4458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:37001
I20260810 22:56:51.401005 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.78.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.189.keytab.
Entry for principal oryx/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.189.keytab.
I20260810 22:56:51.426703 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:46823
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.189.keytab
--principal=oryx/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42665,127.29.78.189:46823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:51.529917 4979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:51.530167 4979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:51.530238 4979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:51.534034 4979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:51.534098 4979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:51.534133 4979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:51.534188 4979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:51.534221 4979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:51.539070 4979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42665,127.29.78.189:46823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.189.keytab
--principal=oryx/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:46823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
W20260810 22:56:51.539263 4979 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.29.78.190:42665,127.29.78.189:46823'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 22:56:51.540067 4979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:51.540966 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:51.548799 4989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.548830 4985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.548827 4986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:51.549204 4979 server_base.cc:1034] running on GCE node
I20260810 22:56:51.549780 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:51.550833 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:51.552004 4979 hybrid_clock.cc:648] HybridClock initialized: now 1786402611551972 us; error 41 us; skew 500 ppm
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:51.554628 4979 init.cc:377] Logged in from keytab as oryx/127.29.78.189@KRBTEST.COM (short username oryx)
I20260810 22:56:51.555377 4979 webserver.cc:460] Webserver started at http://127.29.78.189:39619/ using document root <none> and password file <none>
I20260810 22:56:51.555866 4979 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:51.555923 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:51.556088 4979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:51.557961 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/instance:
uuid: "877d1adeeadf48ee8afa03454a2cc0d0"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "b883f3533ab397a19fb4423513b97159"
server_key_iv: "c0fc53e8ba2a272efa70e09bed20c1d9"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.558439 4979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal/instance:
uuid: "877d1adeeadf48ee8afa03454a2cc0d0"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "b883f3533ab397a19fb4423513b97159"
server_key_iv: "c0fc53e8ba2a272efa70e09bed20c1d9"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.561653 4979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:56:51.563719 4996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.564921 4979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:51.565028 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
uuid: "877d1adeeadf48ee8afa03454a2cc0d0"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "b883f3533ab397a19fb4423513b97159"
server_key_iv: "c0fc53e8ba2a272efa70e09bed20c1d9"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.565107 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:51.593952 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:51.597024 4979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:51.597163 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:51.603642 4979 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:46823
I20260810 22:56:51.603911 5053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:46823 every 8 connection(s)
I20260810 22:56:51.604512 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
I20260810 22:56:51.607123 5054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:51.610798 5054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:56:51.612236 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 4979
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260810 22:56:51.623695 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.612509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.29.78.190:42665)
0810 22:56:51.612663 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.612668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.613371 (+ 703us) server_negotiation.cc:407] Connection header received
0810 22:56:51.614424 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.614426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.614468 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.614528 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.615355 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.615911 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.617006 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.617267 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.617345 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.619895 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.619898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.619900 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.619939 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.621585 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.622159 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.622162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.622164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.622280 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.622624 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.622627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.622628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.622844 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.622950 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.623408 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.623521 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:56:51.624475 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.612820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:56040)
0810 22:56:51.613241 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.613284 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.614225 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.614591 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.614596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.614970 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.615201 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.615236 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.616037 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.616040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.616896 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.616905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.617403 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.617407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.617613 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.618196 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.618210 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.619736 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.621684 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.621690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.621697 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.622070 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.622358 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.622362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.622364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.622511 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.622945 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.622951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.623178 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.623339 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.623634 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"spinlock_wait_cycles":3968,"thread_start_us":111,"threads_started":1}
I20260810 22:56:51.624475 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.613188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.29.78.189:46823)
0810 22:56:51.613571 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.613578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.613848 (+ 270us) server_negotiation.cc:407] Connection header received
0810 22:56:51.614225 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.614247 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.614455 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.614626 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.615282 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.616280 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.617120 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.617549 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.617675 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.620381 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.620399 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.620412 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.620438 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.622162 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.622680 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.622686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.622687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.622770 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.623051 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.623054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.623056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.623277 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.623486 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.623733 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.624129 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"spinlock_wait_cycles":1408,"thread_start_us":126,"threads_started":1}
I20260810 22:56:51.625862 5054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260810 22:56:51.632851 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.626311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.0.0.1:40042)
0810 22:56:51.626408 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.626419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.626493 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.626732 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.626735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.626891 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.626970 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.626974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.627638 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.627640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.628142 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.628149 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.628569 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.628571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.628642 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.629073 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.629086 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.630258 (+ 1172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.631605 (+ 1347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.631607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.631609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.631804 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.632030 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.632032 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.632033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.632102 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.632467 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.632469 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.632539 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.632615 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.632713 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 22:56:51.632899 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.626438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40042 (local address 127.29.78.189:46823)
0810 22:56:51.626553 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.626559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.626571 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:56:51.626601 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.626603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.626632 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.626715 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.627055 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.627535 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.628238 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.628474 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.628547 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.630366 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.630369 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.630370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.630394 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.631509 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.631884 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.631886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.631887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.631951 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.632185 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.632191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.632194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.632401 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.632476 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.632678 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.632757 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 22:56:51.635825 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Bootstrap starting.
I20260810 22:56:51.637992 5054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:51.638708 5054 log.cc:824] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:51.640329 5054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: No bootstrap required, opened a new log
I20260810 22:56:51.643016 5054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } }
I20260810 22:56:51.643184 5054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:51.643254 5054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 877d1adeeadf48ee8afa03454a2cc0d0, State: Initialized, Role: FOLLOWER
I20260810 22:56:51.643678 5054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } }
I20260810 22:56:51.644142 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } } }
I20260810 22:56:51.644258 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:56:51.644752 5054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:56:51.649080 5071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:56:51.649156 5071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:56:51.651167 4979 master_runner.cc:428] Detected that this master 877d1adeeadf48ee8afa03454a2cc0d0 is joining an existing cluster
I20260810 22:56:51.651223 4979 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.78.189:46823
I20260810 22:56:51.662529 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.653518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.0.0.1:40056)
0810 22:56:51.654104 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.654117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.654225 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.654466 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.654468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.654767 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.654965 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.654973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.655984 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.655987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.656942 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.656952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.657394 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.657397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.657525 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.658621 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.658646 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.659095 (+ 449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.661021 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.661024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.661026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.661258 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.661538 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.661540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.661542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.661624 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.662015 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.662017 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.662104 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.662202 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.662317 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":64,"thread_start_us":117,"threads_started":1}
I20260810 22:56:51.662539 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.653646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.29.78.189:46823)
0810 22:56:51.653761 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.653766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.654205 (+ 439us) server_negotiation.cc:407] Connection header received
0810 22:56:51.654309 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.654312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.654352 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.654424 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.655096 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.655870 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.657068 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.657273 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.657383 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.659226 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.659229 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.659232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.659269 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.660906 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.661354 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.661357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.661360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.661445 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.661709 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.661712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.661714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.661936 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.662029 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.662261 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.662374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":44,"spinlock_wait_cycles":7552}
I20260810 22:56:51.663345 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.653588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56054 (local address 127.29.78.190:42665)
0810 22:56:51.653714 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.653719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.654096 (+ 377us) server_negotiation.cc:407] Connection header received
0810 22:56:51.654313 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.654316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.654361 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.654415 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.655087 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.655871 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.658102 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.658526 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.658653 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.659827 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.659833 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.659836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.659864 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.661483 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.661938 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.661942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.661943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.662023 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.662307 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.662312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.662313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.662519 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.662612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.663106 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.663217 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:56:51.663848 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.653512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:56054)
0810 22:56:51.654009 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.654024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.654162 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.654565 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.654567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.654791 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.654953 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.654959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.657095 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.657098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.657972 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.657983 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.658660 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.658664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.658750 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.659270 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.659284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.659662 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.661590 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.661593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.661595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.661814 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.662123 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.662124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.662125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.662209 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.662831 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.662833 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.662919 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.663581 (+ 662us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.663707 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":36,"spinlock_wait_cycles":896,"thread_start_us":105,"threads_started":1}
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260810 22:56:51.676321 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.667993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.29.78.189:46823)
0810 22:56:51.668096 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.668100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.668388 (+ 288us) server_negotiation.cc:407] Connection header received
0810 22:56:51.668646 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.668649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.668695 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.668762 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.669540 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.669981 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.671056 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.671225 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.671291 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.673395 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.673403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.673404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.673422 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.674739 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.675153 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.675158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.675159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.675228 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.675476 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.675480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.675482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.675626 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.675760 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.676084 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.676189 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 22:56:51.676360 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.667969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.0.0.1:40070)
0810 22:56:51.668276 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.668326 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.668473 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.668816 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.668821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.669275 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.669388 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.669397 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.670073 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.670076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.670958 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.670968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.671316 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.671319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.671421 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.671851 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.671866 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.673262 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.674838 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.674847 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.674852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.675062 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.675317 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.675319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.675321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.675397 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.675776 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.675783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.675892 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.676029 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.676203 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260810 22:56:51.677740 4458 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.78.189:46823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0"
member_type: NON_VOTER
last_known_addr {
host: "127.29.78.189"
port: 46823
}
attrs {
promote: true
}
}
dest_uuid: "6318f97f3e634a06ab2be1f8cddbcecb"
cas_config_opid_index: -1
I20260810 22:56:51.678495 4458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } }
I20260810 22:56:51.680161 5083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 877d1adeeadf48ee8afa03454a2cc0d0 (127.29.78.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } }
I20260810 22:56:51.680975 5026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:51.682137 5083 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.78.189:46823
I20260810 22:56:51.682436 5026 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Refusing update from remote peer 6318f97f3e634a06ab2be1f8cddbcecb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 22:56:51.682432 5082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } }
I20260810 22:56:51.682543 5082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:51.682775 4979 master.cc:537] Master@127.29.78.189:46823 shutting down...
I20260810 22:56:51.683233 5083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 22:56:51.685531 4440 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.29.78.188@KRBTEST.COM; defaulting to no policy
W20260810 22:56:51.690372 4440 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb -> Peer 877d1adeeadf48ee8afa03454a2cc0d0 (127.29.78.189:46823): Couldn't send request to peer 877d1adeeadf48ee8afa03454a2cc0d0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:56:51.690455 4979 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:51.690698 4979 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:51.690801 4979 tablet_replica.cc:330] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.188@KRBTEST.COM" created.
I20260810 22:56:51.696753 4979 master.cc:552] Master@127.29.78.189:46823 shutdown complete.
I20260810 22:56:51.696879 4979 master_runner.cc:305] Clearing existing system tablet
I20260810 22:56:51.697152 4979 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:51.697238 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:51.698385 4979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:51.699196 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.699360 4979 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:51.699573 4979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:51.699638 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
uuid: "877d1adeeadf48ee8afa03454a2cc0d0"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "b883f3533ab397a19fb4423513b97159"
server_key_iv: "c0fc53e8ba2a272efa70e09bed20c1d9"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.701443 4979 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:51.703649 4979 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 22:56:51.703794 4979 log.cc:1196] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 22:56:51.704078 4979 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Deleting consensus metadata
I20260810 22:56:51.704280 4979 master_runner.cc:315] Copying system tablet from 127.29.78.190:42665
I20260810 22:56:51.705777 4979 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: tablet copy: Beginning tablet copy session from remote peer at address 127.29.78.190:42665
I20260810 22:56:51.715307 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.706600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.29.78.190:42665)
0810 22:56:51.706739 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.706746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.706892 (+ 146us) server_negotiation.cc:407] Connection header received
0810 22:56:51.707028 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.707031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.707087 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.707164 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.708516 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.709360 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.710545 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.710829 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.710943 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.712145 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.712152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.712158 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.712188 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.713654 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.714201 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.714204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.714205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.714279 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.714534 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.714535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.714536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.714743 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.714817 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.715119 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.715206 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:56:51.715360 5094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.706487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:56058)
0810 22:56:51.706777 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.706805 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.706928 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.707188 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.707191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.707380 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.708381 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.708390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.709518 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.709521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.710419 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.710429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.710965 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.710969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.711093 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.711680 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.711693 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.712010 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.713755 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.713759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.713761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.714099 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.714360 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.714362 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.714364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.714453 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.714822 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.714825 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.714923 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.715063 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.715188 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260810 22:56:51.716181 4478 tablet_copy_service.cc:140] P 6318f97f3e634a06ab2be1f8cddbcecb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 877d1adeeadf48ee8afa03454a2cc0d0 ({username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:56058)
I20260810 22:56:51.716348 4478 tablet_copy_service.cc:161] P 6318f97f3e634a06ab2be1f8cddbcecb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 877d1adeeadf48ee8afa03454a2cc0d0 at {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:56058: session id = 877d1adeeadf48ee8afa03454a2cc0d0-00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.188.keytab.
Entry for principal oryx/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.188.keytab.
I20260810 22:56:51.717959 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:44697
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.188.keytab
--principal=oryx/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:56:51.718128 4478 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:56:51.719627 4979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 22:56:51.722448 4979 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: tablet copy: Starting download of 0 data blocks...
I20260810 22:56:51.722676 4979 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: tablet copy: Starting download of 1 WAL segments...
I20260810 22:56:51.724412 4979 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:56:51.726511 4478 tablet_copy_service.cc:342] P 6318f97f3e634a06ab2be1f8cddbcecb: Request end of tablet copy session 877d1adeeadf48ee8afa03454a2cc0d0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:56058
I20260810 22:56:51.726639 4478 tablet_copy_service.cc:433] P 6318f97f3e634a06ab2be1f8cddbcecb: ending tablet copy session 877d1adeeadf48ee8afa03454a2cc0d0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 877d1adeeadf48ee8afa03454a2cc0d0
I20260810 22:56:51.734856 4979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:51.736397 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.736392 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.736725 5100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:51.736944 4979 server_base.cc:1034] running on GCE node
I20260810 22:56:51.737185 4979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:51.737520 4979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:51.738700 4979 hybrid_clock.cc:648] HybridClock initialized: now 1786402611738687 us; error 27 us; skew 500 ppm
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:51.740411 4979 init.cc:377] Logged in from keytab as oryx/127.29.78.189@KRBTEST.COM (short username oryx)
I20260810 22:56:51.740893 4979 webserver.cc:460] Webserver started at http://127.29.78.189:39639/ using document root <none> and password file <none>
I20260810 22:56:51.741144 4979 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:51.741233 4979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:51.742507 4979 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:51.743476 5107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.743705 4979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:51.743805 4979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
uuid: "877d1adeeadf48ee8afa03454a2cc0d0"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "b883f3533ab397a19fb4423513b97159"
server_key_iv: "c0fc53e8ba2a272efa70e09bed20c1d9"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.743927 4979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:51.767807 4979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:51.770871 4979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:51.776821 4979 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:46823
I20260810 22:56:51.776886 5164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:46823 every 8 connection(s)
I20260810 22:56:51.777575 4979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
I20260810 22:56:51.779554 5165 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:56:51.780607 5165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Bootstrap starting.
I20260810 22:56:51.790061 5165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:56:51.790324 5165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Bootstrap complete.
I20260810 22:56:51.790792 5165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } }
I20260810 22:56:51.790930 5165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 877d1adeeadf48ee8afa03454a2cc0d0, State: Initialized, Role: LEARNER
I20260810 22:56:51.791021 5165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } }
I20260810 22:56:51.791303 5168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } }
I20260810 22:56:51.791415 5165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:51.791496 5168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: LEARNER
I20260810 22:56:51.794672 5180 catalog_manager.cc:1259] Loaded cluster ID: 2a5f72c4338d4d55b84ee7c2c4923a66
I20260810 22:56:51.794723 5180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: loading cluster ID for follower catalog manager: success
I20260810 22:56:51.796643 5180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: acquiring CA information for follower catalog manager: success
I20260810 22:56:51.797866 5180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 22:56:51.844645 5095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:51.844892 5095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:51.844929 5095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:51.848719 5095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:51.848764 5095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:51.848781 5095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:51.848795 5095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:51.848809 5095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:51.853633 5095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.188.keytab
--principal=oryx/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:51.854571 5095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:51.855517 5095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:51.863215 5187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:51.863323 5095 server_base.cc:1034] running on GCE node
W20260810 22:56:51.863206 5190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:51.863209 5186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:51.864013 5095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:51.864984 5095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:51.866133 5095 hybrid_clock.cc:648] HybridClock initialized: now 1786402611866115 us; error 27 us; skew 500 ppm
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:51.868721 5095 init.cc:377] Logged in from keytab as oryx/127.29.78.188@KRBTEST.COM (short username oryx)
I20260810 22:56:51.869494 5095 webserver.cc:460] Webserver started at http://127.29.78.188:38795/ using document root <none> and password file <none>
I20260810 22:56:51.870110 5095 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:51.870196 5095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:51.870409 5095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:51.872391 5095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/instance:
uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "08f20b3538b52943b5f562c31e4c60eb"
server_key_iv: "f9fad0b07842df574df512287a261f11"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.872960 5095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal/instance:
uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "08f20b3538b52943b5f562c31e4c60eb"
server_key_iv: "f9fad0b07842df574df512287a261f11"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.876725 5095 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 22:56:51.879073 5198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:51.880240 5095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:51.880370 5095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "08f20b3538b52943b5f562c31e4c60eb"
server_key_iv: "f9fad0b07842df574df512287a261f11"
server_key_version: "encryptionkey@0"
I20260810 22:56:51.880471 5095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:51.912742 5095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:51.915498 5095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:51.915711 5095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:51.923430 5095 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:44697
I20260810 22:56:51.923715 5254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:44697 every 8 connection(s)
I20260810 22:56:51.924368 5095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
I20260810 22:56:51.925055 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 5095
I20260810 22:56:51.927445 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:51.931968 5255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:51.940222 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.926509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40362 (local address 127.29.78.188:44697)
0810 22:56:51.926931 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.926948 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.926986 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:56:51.927765 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.927786 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.928171 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.928497 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.929152 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.930620 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.931455 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.932144 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.932300 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.935215 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.935219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.935228 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.935254 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.936988 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.937551 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.937556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.937561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.937688 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.938085 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.938090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.938093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.938455 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.938674 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.938955 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.939250 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260810 22:56:51.943418 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.934026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.29.78.190:42665)
0810 22:56:51.934160 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.934164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.934442 (+ 278us) server_negotiation.cc:407] Connection header received
0810 22:56:51.934575 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.934580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.934647 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.934720 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.935564 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.936416 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.937471 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.937833 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.937933 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.940270 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.940273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.940288 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.940319 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.941788 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.942219 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.942222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.942224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.942316 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.942593 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.942598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.942600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.942781 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.942879 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.943168 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.943273 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:56:51.943459 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.933874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:56072)
0810 22:56:51.934304 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.934357 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.934477 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.934779 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.934787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.935278 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.935412 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.935418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.936532 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.936535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.937335 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.937346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.937958 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.937962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.938097 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.938607 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.938619 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.940134 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.941931 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.941937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.941941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.942132 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.942410 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.942413 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.942415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.942493 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.942877 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.942883 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.942984 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.943114 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.943273 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":153,"threads_started":1}
I20260810 22:56:51.944348 5255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260810 22:56:51.951502 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.944729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.0.0.1:40084)
0810 22:56:51.944828 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.944841 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.944908 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.945253 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.945255 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.945370 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.945462 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.945466 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.946374 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.946376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.946884 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.946903 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.947328 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.947329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.947395 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.947711 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.947720 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.948739 (+ 1019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.950427 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.950429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.950430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.950611 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.950829 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.950831 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.950832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.950896 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.951183 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.951184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.951248 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.951321 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.951405 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 22:56:51.951665 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.944828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.29.78.189:46823)
0810 22:56:51.945049 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.945052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.945062 (+ 10us) server_negotiation.cc:407] Connection header received
0810 22:56:51.945096 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.945097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.945131 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.945244 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.945557 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.946288 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.946984 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.947241 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.947314 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.948846 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.948848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.948850 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.948871 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.950333 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.950687 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.950689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.950691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.950759 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.950964 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.950966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.950967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.951112 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.951183 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.951381 (+ 198us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.951468 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260810 22:56:51.952253 5255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:56:51 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402611, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
I20260810 22:56:51.959425 5258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.952752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.0.0.1:40372)
0810 22:56:51.952845 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.952854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.952907 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.953080 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.953082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.953193 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.953251 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.953255 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.954290 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.954292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.954714 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.954720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.955075 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.955078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.955126 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.955455 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.955464 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.956487 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.958237 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.958239 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.958240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.958396 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.958641 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.958643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.958644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.958706 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.959131 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.959133 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.959192 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.959261 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.959338 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 22:56:51.959550 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.952789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40372 (local address 127.29.78.188:44697)
0810 22:56:51.952893 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.952896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.952906 (+ 10us) server_negotiation.cc:407] Connection header received
0810 22:56:51.952959 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.952961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.952990 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.953047 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.953335 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.954189 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.954798 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.954991 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.955063 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.956617 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.956620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.956622 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.956643 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.958122 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.958475 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.958477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.958479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.958559 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.958777 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.958779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.958781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.959039 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.959148 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.959334 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.959432 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 22:56:51.962239 5255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Bootstrap starting.
I20260810 22:56:51.964272 5255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:51.964941 5255 log.cc:824] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:51.966790 5255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: No bootstrap required, opened a new log
I20260810 22:56:51.969496 5255 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } }
I20260810 22:56:51.969707 5255 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:51.969779 5255 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb2f666902964b2dbaa5fe0bae8ac1bc, State: Initialized, Role: FOLLOWER
I20260810 22:56:51.970244 5255 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } }
I20260810 22:56:51.970728 5261 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } } }
I20260810 22:56:51.970840 5261 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:56:51.971355 5255 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: configured and running, proceeding with master startup.
W20260810 22:56:51.976060 5272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:56:51.976151 5272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:56:51.979410 5095 master_runner.cc:428] Detected that this master eb2f666902964b2dbaa5fe0bae8ac1bc is joining an existing cluster
I20260810 22:56:51.979461 5095 master_runner.cc:432] Initiating AddMaster RPC to add 127.29.78.188:44697
I20260810 22:56:51.980762 4673 heartbeater.cc:503] Master 127.29.78.190:42665 was elected leader, sending a full tablet report...
I20260810 22:56:51.989722 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.981852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.0.0.1:40378)
0810 22:56:51.982237 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.982251 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.982346 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.982642 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.982645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.982917 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.983056 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.983064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.984140 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.984143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.985055 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.985065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.985638 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.985641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.985779 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.986368 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:51.986383 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:51.986856 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:51.988370 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.988373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.988375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.988615 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.988842 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:51.988845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:51.988847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:51.988932 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:51.989222 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:51.989224 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:51.989303 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 22:56:51.989400 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 22:56:51.989543 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260810 22:56:51.989750 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.981696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40378 (local address 127.29.78.188:44697)
0810 22:56:51.981863 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.981869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.982325 (+ 456us) server_negotiation.cc:407] Connection header received
0810 22:56:51.982441 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.982444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.982479 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.982522 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.983201 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.983991 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.985161 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.985516 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.985624 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:51.987001 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:51.987004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:51.987006 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:51.987025 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:51.988276 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.988701 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.988703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.988704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.988768 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:51.989014 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:51.989016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:51.989016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:51.989152 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:51.989211 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:51.989443 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 22:56:51.989622 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":112,"server-negotiator.queue_time_us":57}
I20260810 22:56:51.993717 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 4527
I20260810 22:56:52.008630 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.982088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40094 (local address 127.29.78.189:46823)
0810 22:56:51.982232 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.982237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.983154 (+ 917us) server_negotiation.cc:407] Connection header received
0810 22:56:51.983339 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.983344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.983410 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.983463 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:51.984135 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.984635 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.985639 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.986025 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.986111 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.003791 (+ 17680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.003795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.003797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.003828 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.006006 (+ 2178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.006557 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.006563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.006568 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.006654 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.007284 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.007289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.007292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.007826 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.007926 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.008304 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.008452 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:52.009222 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.982042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.0.0.1:40094)
0810 22:56:51.983070 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.983082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.983218 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.983604 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.983607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:51.983895 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:51.984016 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.984023 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.984736 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.984741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.985524 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:51.985530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:51.986271 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:51.986276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:51.986388 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.003204 (+ 16816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.003219 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.003631 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.006136 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.006142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.006146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.006432 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.006776 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.006779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.006781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.006908 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.008010 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.008012 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.008117 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.008843 (+ 726us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.009006 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":33,"thread_start_us":153,"threads_started":1}
I20260810 22:56:52.009701 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:36833
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:42665
--webserver_port=42027
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:56:52.010768 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.982779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:56084)
0810 22:56:51.983100 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:51.983114 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:51.983256 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:51.983604 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:51.983607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.002395 (+ 18788us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.002543 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.002556 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.003670 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.003676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.004526 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.004539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.005350 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.005367 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.005476 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.005969 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.005984 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.006465 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.008556 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.008559 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.008561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.008843 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.009158 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.009160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.009162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.009244 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.010248 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.010251 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.010341 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.010460 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.010592 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"spinlock_wait_cycles":14080,"thread_start_us":137,"threads_started":1}
I20260810 22:56:52.010774 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:51.982637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.29.78.190:42665)
0810 22:56:51.982752 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:51.982756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:51.983172 (+ 416us) server_negotiation.cc:407] Connection header received
0810 22:56:51.983378 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:51.983381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:51.983423 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:51.983473 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.002674 (+ 19201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.003527 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.004677 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.005069 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.005178 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.006615 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.006621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.006624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.006649 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.008411 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.008955 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.008960 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.008964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.009049 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.009372 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.009376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.009378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.009601 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.009673 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.010525 (+ 852us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.010636 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402610, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
I20260810 22:56:52.024144 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.014660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.0.0.1:40390)
0810 22:56:52.014780 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.014794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.014874 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.015321 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.015327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.015537 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.015670 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.015697 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.016810 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.016816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.017734 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.017746 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.018323 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.018330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.018450 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.018923 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.018941 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.020567 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.022614 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.022617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.022619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.022869 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.023160 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.023163 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.023165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.023254 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.023634 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.023637 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.023733 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.023844 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.023977 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 22:56:52.024169 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.014793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.29.78.188:44697)
0810 22:56:52.014928 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.014933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.014949 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:56:52.015020 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.015025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.015091 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.015178 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.015839 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.016662 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.017896 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.018204 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.018300 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.020713 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.020717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.020720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.020743 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.022481 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.022971 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.022974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.022976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.023060 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.023345 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.023348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.023349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.023541 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.023634 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.023910 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.024025 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 22:56:52.025573 4458 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.29.78.188:44697: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
member_type: NON_VOTER
last_known_addr {
host: "127.29.78.188"
port: 44697
}
attrs {
promote: true
}
}
dest_uuid: "6318f97f3e634a06ab2be1f8cddbcecb"
cas_config_opid_index: 5
I20260810 22:56:52.026235 4458 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } }
I20260810 22:56:52.027281 5083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.028630 5083 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.29.78.188:44697
I20260810 22:56:52.028790 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.028887 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:52.029422 5095 master.cc:537] Master@127.29.78.188:44697 shutting down...
I20260810 22:56:52.029789 5227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:52.031184 5227 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Refusing update from remote peer 6318f97f3e634a06ab2be1f8cddbcecb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 22:56:52.031865 5261 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:52.031822 5082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:52.032020 5261 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } }
I20260810 22:56:52.033296 5261 leader_election.cc:288] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665), 877d1adeeadf48ee8afa03454a2cc0d0 (127.29.78.189:46823)
I20260810 22:56:52.033741 5136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" is_pre_election: true
I20260810 22:56:52.033730 4468 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" is_pre_election: true
I20260810 22:56:52.033962 5136 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 LEARNER]: Handling vote request from an unknown peer eb2f666902964b2dbaa5fe0bae8ac1bc
I20260810 22:56:52.034085 5136 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate eb2f666902964b2dbaa5fe0bae8ac1bc for term 2 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 22:56:52.034475 5199 leader_election.cc:302] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f666902964b2dbaa5fe0bae8ac1bc; no voters: 6318f97f3e634a06ab2be1f8cddbcecb, 877d1adeeadf48ee8afa03454a2cc0d0
W20260810 22:56:52.035010 4441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb -> Peer eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188:44697): Couldn't send request to peer eb2f666902964b2dbaa5fe0bae8ac1bc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 22:56:52.035384 5261 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:56:52.039363 5095 raft_consensus.cc:2229] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:52.039609 5095 raft_consensus.cc:2258] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:52.039703 5095 tablet_replica.cc:330] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: stopping tablet replica
I20260810 22:56:52.040733 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.027405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.0.0.1:40106)
0810 22:56:52.035827 (+ 8422us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.035843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.035936 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.036298 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.036301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.036318 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:52.036400 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.036405 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.037413 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.037416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.038694 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.038705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.040327 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.040331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.040436 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.040479 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.040551 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.040616 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8344}
I20260810 22:56:52.041147 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.027615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.29.78.189:46823)
0810 22:56:52.027800 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.027805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.035904 (+ 8099us) server_negotiation.cc:407] Connection header received
0810 22:56:52.036020 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.036023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.036084 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.036173 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:52.036519 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.037281 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.038905 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.040022 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.040118 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.040267 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.040970 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.041043 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 22:56:52.041879 5136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 LEARNER]: Refusing update from remote peer 6318f97f3e634a06ab2be1f8cddbcecb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 22:56:52.042302 5082 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 22:56:52.044134 5136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.044816 5095 master.cc:552] Master@127.29.78.188:44697 shutdown complete.
I20260810 22:56:52.044854 5095 master_runner.cc:305] Clearing existing system tablet
I20260810 22:56:52.045094 5095 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:52.045138 5095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:52.046468 5095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 22:56:52.046792 5168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6318f97f3e634a06ab2be1f8cddbcecb. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.046924 5168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: LEARNER
I20260810 22:56:52.047010 5168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: NON_VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: true } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.047070 5168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: LEARNER
I20260810 22:56:52.047253 5286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.047432 5095 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:52.047531 5095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 22:56:52.047561 5095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "08f20b3538b52943b5f562c31e4c60eb"
server_key_iv: "f9fad0b07842df574df512287a261f11"
server_key_version: "encryptionkey@0"
I20260810 22:56:52.049415 5095 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:52.051630 5095 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 22:56:52.051735 5095 log.cc:1196] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 22:56:52.051978 5095 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Deleting consensus metadata
I20260810 22:56:52.052150 5095 master_runner.cc:315] Copying system tablet from 127.29.78.190:42665
I20260810 22:56:52.053220 5095 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: tablet copy: Beginning tablet copy session from remote peer at address 127.29.78.190:42665
I20260810 22:56:52.055008 5082 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: attempting to promote NON_VOTER 877d1adeeadf48ee8afa03454a2cc0d0 to VOTER
I20260810 22:56:52.055444 5082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } }
I20260810 22:56:52.056818 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.056540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.0.0.1:40402)
0810 22:56:52.056678 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.056743 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 22:56:52.057323 5136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 LEARNER]: Refusing update from remote peer 6318f97f3e634a06ab2be1f8cddbcecb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 22:56:52.057691 5083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 22:56:52.057811 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.057580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.0.0.1:40404)
0810 22:56:52.057699 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.057743 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:44697: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 22:56:52.057958 4441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb -> Peer eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188:44697): Couldn't send request to peer eb2f666902964b2dbaa5fe0bae8ac1bc. Status: Network error: Client connection negotiation failed: client connection to 127.29.78.188:44697: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:56:52.059372 5083 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 877d1adeeadf48ee8afa03454a2cc0d0 (127.29.78.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.060658 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.060750 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:52.060855 5083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.060914 5083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:52.061618 5136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 877d1adeeadf48ee8afa03454a2cc0d0 (127.29.78.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.063131 5284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.063259 5284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:56:52.065007 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.053757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56092 (local address 127.29.78.190:42665)
0810 22:56:52.053883 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.053887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.054097 (+ 210us) server_negotiation.cc:407] Connection header received
0810 22:56:52.054193 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.054196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.054248 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.054355 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.054902 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.055671 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.058103 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.058406 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.058477 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.061348 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.061352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.061354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.061379 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.063112 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.063718 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.063722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.063724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.063827 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.064175 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.064180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.064181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.064376 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.064442 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.064769 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.064887 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 22:56:52.065665 5293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.053683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:56092)
0810 22:56:52.053999 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.054013 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.054117 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.054386 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.054391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.054631 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.054750 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.054757 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.056256 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.056261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.057221 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.057232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.058628 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.058632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.060127 (+ 1495us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.060701 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.060717 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.061200 (+ 483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.063335 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.063339 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.063341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.063589 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.063962 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.063965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.063967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.064069 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.064512 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.064515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.064605 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.065276 (+ 671us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.065416 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":76,"threads_started":1}
I20260810 22:56:52.066337 4478 tablet_copy_service.cc:140] P 6318f97f3e634a06ab2be1f8cddbcecb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer eb2f666902964b2dbaa5fe0bae8ac1bc ({username='oryx', principal='oryx/127.29.78.188@KRBTEST.COM'} at 127.0.0.1:56092)
I20260810 22:56:52.066447 4478 tablet_copy_service.cc:161] P 6318f97f3e634a06ab2be1f8cddbcecb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer eb2f666902964b2dbaa5fe0bae8ac1bc at {username='oryx', principal='oryx/127.29.78.188@KRBTEST.COM'} at 127.0.0.1:56092: session id = eb2f666902964b2dbaa5fe0bae8ac1bc-00000000000000000000000000000000
I20260810 22:56:52.067615 4478 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 22:56:52.068827 5095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 22:56:52.070977 5095 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: tablet copy: Starting download of 0 data blocks...
I20260810 22:56:52.071162 5095 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: tablet copy: Starting download of 1 WAL segments...
I20260810 22:56:52.072723 5095 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 22:56:52.074491 4478 tablet_copy_service.cc:342] P 6318f97f3e634a06ab2be1f8cddbcecb: Request end of tablet copy session eb2f666902964b2dbaa5fe0bae8ac1bc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.29.78.188@KRBTEST.COM'} at 127.0.0.1:56092
I20260810 22:56:52.074569 4478 tablet_copy_service.cc:433] P 6318f97f3e634a06ab2be1f8cddbcecb: ending tablet copy session eb2f666902964b2dbaa5fe0bae8ac1bc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer eb2f666902964b2dbaa5fe0bae8ac1bc
I20260810 22:56:52.082144 5095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:52.083806 5295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:52.084039 5095 server_base.cc:1034] running on GCE node
W20260810 22:56:52.084157 5296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:52.084290 5298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:52.084566 5095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:52.084863 5095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:52.086004 5095 hybrid_clock.cc:648] HybridClock initialized: now 1786402612085982 us; error 34 us; skew 500 ppm
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:52.087626 5095 init.cc:377] Logged in from keytab as oryx/127.29.78.188@KRBTEST.COM (short username oryx)
I20260810 22:56:52.088078 5095 webserver.cc:460] Webserver started at http://127.29.78.188:35851/ using document root <none> and password file <none>
I20260810 22:56:52.088258 5095 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:52.088315 5095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:52.089537 5095 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:52.090498 5306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.090751 5095 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 22:56:52.090819 5095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "08f20b3538b52943b5f562c31e4c60eb"
server_key_iv: "f9fad0b07842df574df512287a261f11"
server_key_version: "encryptionkey@0"
I20260810 22:56:52.090895 5095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:52.120646 5095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:52.123059 5095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:52.128218 5095 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:44697
I20260810 22:56:52.128249 5365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:44697 every 8 connection(s)
I20260810 22:56:52.128809 5095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
I20260810 22:56:52.130765 5366 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:56:52.131901 5366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Bootstrap starting.
I20260810 22:56:52.143481 5366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:56:52.143756 5366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: Bootstrap complete.
I20260810 22:56:52.144413 5366 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } }
I20260810 22:56:52.144546 5366 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eb2f666902964b2dbaa5fe0bae8ac1bc, State: Initialized, Role: LEARNER
I20260810 22:56:52.144644 5366 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } }
I20260810 22:56:52.145061 5366 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:52.145373 5372 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.145609 5372 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: This master's current role is: LEARNER
I20260810 22:56:52.149582 5386 catalog_manager.cc:1259] Loaded cluster ID: 2a5f72c4338d4d55b84ee7c2c4923a66
I20260810 22:56:52.149645 5386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: loading cluster ID for follower catalog manager: success
W20260810 22:56:52.151029 5282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:52.151291 5282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:52.151373 5282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 22:56:52.151822 5386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: acquiring CA information for follower catalog manager: success
I20260810 22:56:52.153252 5386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 22:56:52.155615 5282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:52.155689 5282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:52.155814 5282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:52.161055 5282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:36833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=42027
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:52.162128 5282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:52.163098 5282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:52.173038 5389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:52.173035 5388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:52.173032 5391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:52.173334 5282 server_base.cc:1034] running on GCE node
I20260810 22:56:52.173686 5282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:52.174299 5282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:52.175455 5282 hybrid_clock.cc:648] HybridClock initialized: now 1786402612175438 us; error 26 us; skew 500 ppm
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:52.178092 5282 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260810 22:56:52.178841 5282 webserver.cc:460] Webserver started at http://127.29.78.129:42027/ using document root <none> and password file <none>
I20260810 22:56:52.179347 5282 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:52.179437 5282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:52.183014 5282 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:56:52.184875 5398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.185992 5282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:52.186111 5282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "5fd6b2a2130d427c8bbe156ed85f7d06"
format_stamp: "Formatted at 2026-08-10 22:56:50 on dist-test-slave-6nmv"
server_key: "e0ae2fd19a315d0daff6bc80c197fac5"
server_key_iv: "ca023d3081a0ab0064b0b3bc83021eb7"
server_key_version: "encryptionkey@0"
I20260810 22:56:52.186622 5282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:52.188830 4822 heartbeater.cc:503] Master 127.29.78.190:42665 was elected leader, sending a full tablet report...
I20260810 22:56:52.202664 5282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:52.205374 5282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:52.205531 5282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:52.206147 5282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:52.206998 5282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:52.207049 5282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.207093 5282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:52.207116 5282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.217571 5282 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:36833
I20260810 22:56:52.217588 5522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:36833 every 8 connection(s)
I20260810 22:56:52.218533 5282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:52.218755 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 5282
I20260810 22:56:52.218958 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 4679
I20260810 22:56:52.228883 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:42085
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:42665
--webserver_port=43137
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260810 22:56:52.237406 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.221287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:38677 (local address 127.29.78.188:44697)
0810 22:56:52.221774 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.221779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.223198 (+ 1419us) server_negotiation.cc:407] Connection header received
0810 22:56:52.223688 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.223693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.223755 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.223817 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.226017 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.227214 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.228801 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.229169 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.229254 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.233030 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.233033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.233036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.233063 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.234931 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.235546 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.235550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.235552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.235639 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.236078 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.236083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.236088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.236320 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.236409 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.237094 (+ 685us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.237222 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":104,"threads_started":1}
I20260810 22:56:52.237671 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.221438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:60127 (local address 127.29.78.190:42665)
0810 22:56:52.221606 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.221612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.222381 (+ 769us) server_negotiation.cc:407] Connection header received
0810 22:56:52.223705 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.223708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.223771 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.223828 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.226018 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.226854 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.228799 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.229120 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.229200 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.233627 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.233633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.233638 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.233674 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.235920 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.236398 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.236401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.236402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.236490 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.236801 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.236804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.236806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.237051 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.237121 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.237420 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.237547 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:56:52.238744 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.224289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:48795 (local address 127.29.78.189:46823)
0810 22:56:52.224443 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.224449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.224859 (+ 410us) server_negotiation.cc:407] Connection header received
0810 22:56:52.225078 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.225082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.225150 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.225206 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.226017 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.226855 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.228799 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.229129 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.229200 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.234022 (+ 4822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.234028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.234032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.234067 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.236069 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.236723 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.236730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.236734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.236838 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.237246 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.237250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.237252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.237508 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.237594 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.238403 (+ 809us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.238535 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:56:52.238896 5529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.221287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:60127)
0810 22:56:52.222158 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.222245 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.223450 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.224199 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.224207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.224953 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.225829 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.225852 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.227071 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.227075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.228652 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.228663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.229419 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.229423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.229762 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.230708 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.230743 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.233482 (+ 2739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.236042 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.236048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.236053 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.236280 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.236610 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.236613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.236615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.236695 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.237169 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.237171 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.237249 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.237868 (+ 619us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.237966 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":434,"spinlock_wait_cycles":26240,"thread_start_us":135,"threads_started":1}
I20260810 22:56:52.238894 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.221287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.29.78.129:38677)
0810 22:56:52.223077 (+ 1790us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.223091 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.223464 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.224200 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.224209 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.224956 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.225832 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.225858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.227354 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.227359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.228671 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.228684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.229734 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.229739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.229923 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.230847 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.230862 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.232828 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.235076 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.235083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.235097 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.235435 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.235771 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.235777 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.235781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.235954 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.236475 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.236481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.236787 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.237552 (+ 765us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.237868 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1696,"mutex_wait_us":429,"thread_start_us":125,"threads_started":1}
I20260810 22:56:52.240119 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.223900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.29.78.129:48795)
0810 22:56:52.224750 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.224765 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.224937 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.225352 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.225355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.225545 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.225844 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.225852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.227071 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.227075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.228655 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.228663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.229419 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.229423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.229763 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.230734 (+ 971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.230751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.233852 (+ 3101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.236230 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.236236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.236239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.236578 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.237012 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.237017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.237020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.237113 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.238063 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.238067 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.238176 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.239728 (+ 1552us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.239872 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":364,"spinlock_wait_cycles":3840,"thread_start_us":105,"threads_started":1}
I20260810 22:56:52.240145 5523 heartbeater.cc:347] Connected to a master server at 127.29.78.188:44697
I20260810 22:56:52.240170 5524 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42665
I20260810 22:56:52.240650 5524 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.240650 5523 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.241255 5523 heartbeater.cc:511] Master 127.29.78.188:44697 requested a full tablet report, sending...
I20260810 22:56:52.241255 5524 heartbeater.cc:511] Master 127.29.78.190:42665 requested a full tablet report, sending...
I20260810 22:56:52.241353 5525 heartbeater.cc:347] Connected to a master server at 127.29.78.189:46823
I20260810 22:56:52.241437 5525 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.241631 5525 heartbeater.cc:511] Master 127.29.78.189:46823 requested a full tablet report, sending...
I20260810 22:56:52.242429 4458 ts_manager.cc:194] Re-registered known tserver with Master: 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129:36833)
I20260810 22:56:52.242678 5328 ts_manager.cc:194] Registered new tserver with Master: 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129:36833)
I20260810 22:56:52.242868 5126 ts_manager.cc:194] Registered new tserver with Master: 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129:36833)
I20260810 22:56:52.243108 4458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:60127
W20260810 22:56:52.339445 5533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:52.339687 5533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:52.339756 5533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:52.343539 5533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:52.343611 5533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:52.343693 5533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:52.348860 5533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:42085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=43137
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:52.349898 5533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:52.350837 5533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:52.359714 5539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:52.359721 5541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:52.359710 5538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:52.359932 5533 server_base.cc:1034] running on GCE node
I20260810 22:56:52.360302 5533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:52.360847 5533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:52.361989 5533 hybrid_clock.cc:648] HybridClock initialized: now 1786402612361971 us; error 29 us; skew 500 ppm
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:52.364500 5533 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260810 22:56:52.365243 5533 webserver.cc:460] Webserver started at http://127.29.78.130:43137/ using document root <none> and password file <none>
I20260810 22:56:52.365765 5533 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:52.365897 5533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:52.369622 5533 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 22:56:52.371416 5548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.372573 5533 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 22:56:52.372663 5533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "38f13c2dd9314170b397a16a4a650007"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "8c8bac140dc118d42100bbaa321e9c4d"
server_key_iv: "51a40ac54cf98a3ba463924ee48891f1"
server_key_version: "encryptionkey@0"
I20260810 22:56:52.373147 5533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:52.390733 5533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:52.393188 4974 heartbeater.cc:503] Master 127.29.78.190:42665 was elected leader, sending a full tablet report...
I20260810 22:56:52.394264 5533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:52.394475 5533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:52.395017 5533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:52.395917 5533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:52.395998 5533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.396082 5533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:52.396133 5533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.404899 5533 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:42085
I20260810 22:56:52.404913 5672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:42085 every 8 connection(s)
I20260810 22:56:52.406075 5533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:52.416352 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 5533
I20260810 22:56:52.416541 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 4827
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260810 22:56:52.426797 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.408811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:50553 (local address 127.29.78.188:44697)
0810 22:56:52.408949 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.408953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.409640 (+ 687us) server_negotiation.cc:407] Connection header received
0810 22:56:52.410540 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.410544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.410613 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.410689 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.411970 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.412810 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414324 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.414560 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414620 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.421851 (+ 7231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.421855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.421858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.421888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.424267 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.424892 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.424898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.424903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.424989 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.425370 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.425374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.425377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.425619 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.425704 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.426461 (+ 757us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.426598 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 22:56:52.426903 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.407887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49311 (local address 127.29.78.190:42665)
0810 22:56:52.408020 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.408024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.409501 (+ 1477us) server_negotiation.cc:407] Connection header received
0810 22:56:52.410534 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.410537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.410592 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.410642 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.411969 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.412810 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414337 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.414684 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414769 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.422151 (+ 7382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.422156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.422160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.422247 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.424165 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.424847 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.424853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.424855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.424946 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.425372 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.425378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.425382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.425620 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.425703 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.426500 (+ 797us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.426643 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:56:52.428442 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.408710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.29.78.130:50553)
0810 22:56:52.409520 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.409532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.410326 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.410864 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.410872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.411276 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.411825 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.411835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.412952 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.412956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.414164 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.414174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.414781 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.415023 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.415793 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.415810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.421650 (+ 5840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.424410 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.424416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.424421 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.424759 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.425114 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.425117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.425119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.425256 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.425810 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.425815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.426171 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.427079 (+ 908us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.427397 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":314,"spinlock_wait_cycles":31360,"thread_start_us":94,"threads_started":1}
I20260810 22:56:52.429899 5673 heartbeater.cc:347] Connected to a master server at 127.29.78.188:44697
I20260810 22:56:52.429946 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.408442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.29.78.130:59249)
0810 22:56:52.409864 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.409877 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.410337 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.410865 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.410872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.411245 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.411811 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.411825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.412637 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.412639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.414144 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.414153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.414775 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.415003 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.415906 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.415921 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.426610 (+ 10689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.428532 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.428535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.428537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.428746 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.428993 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.428997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.428999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.429076 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.429436 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.429438 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.429530 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.429618 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.429762 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":180,"spinlock_wait_cycles":3328,"thread_start_us":725,"threads_started":1}
I20260810 22:56:52.429988 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.408030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:59249 (local address 127.29.78.189:46823)
0810 22:56:52.408180 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.408184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.409954 (+ 1770us) server_negotiation.cc:407] Connection header received
0810 22:56:52.410625 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.410628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.410682 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.410745 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.411956 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.412491 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414283 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.414605 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414733 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.426772 (+ 12039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.426775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.426778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.426810 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.428387 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.428833 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.428836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.428838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.428907 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.429164 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.429167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.429169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.429348 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.429443 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.429682 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.429810 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 22:56:52.430133 5673 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.430480 5676 heartbeater.cc:347] Connected to a master server at 127.29.78.189:46823
I20260810 22:56:52.430559 5676 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.430714 5676 heartbeater.cc:511] Master 127.29.78.189:46823 requested a full tablet report, sending...
I20260810 22:56:52.430716 5673 heartbeater.cc:511] Master 127.29.78.188:44697 requested a full tablet report, sending...
I20260810 22:56:52.428447 5680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.408442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:49311)
0810 22:56:52.409351 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.409404 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.410337 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.410864 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.410872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.411277 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.411829 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.411835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.412952 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.412956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.414164 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.414174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.414896 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.414899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.415026 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.415530 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.415544 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.422022 (+ 6478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.424289 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.424297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.424322 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.424716 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.425055 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.425061 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.425065 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.425249 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.425812 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.425818 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.426173 (+ 355us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.427441 (+ 1268us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.427593 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":490,"spinlock_wait_cycles":46720,"thread_start_us":123,"threads_started":1}
I20260810 22:56:52.431820 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:37281
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:42665
--webserver_port=45575
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:56:52.432178 5328 ts_manager.cc:194] Registered new tserver with Master: 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085)
I20260810 22:56:52.433372 5674 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42665
I20260810 22:56:52.433468 5674 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.433625 5674 heartbeater.cc:511] Master 127.29.78.190:42665 requested a full tablet report, sending...
I20260810 22:56:52.434137 5126 ts_manager.cc:194] Registered new tserver with Master: 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085)
I20260810 22:56:52.434314 4458 ts_manager.cc:194] Re-registered known tserver with Master: 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085)
I20260810 22:56:52.434896 4458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:49311
W20260810 22:56:52.540637 5682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:52.540885 5682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:52.540977 5682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:52.544835 5682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:52.544917 5682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:52.545017 5682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:52.550436 5682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:37281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=45575
--tserver_master_addrs=127.29.78.190:42665,127.29.78.189:46823,127.29.78.188:44697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:52.551451 5682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:52.552469 5682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:52.561667 5689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:52.561668 5688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:52.561808 5682 server_base.cc:1034] running on GCE node
W20260810 22:56:52.561736 5691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:52.562454 5682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:52.563084 5682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:52.566372 5682 hybrid_clock.cc:648] HybridClock initialized: now 1786402612566045 us; error 335 us; skew 500 ppm
I20260810 22:56:52.566756 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.563396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.29.78.188:44697)
0810 22:56:52.563625 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.563629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.563646 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:56:52.563702 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.563704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.563757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.563830 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:52.564071 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.564637 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.565634 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.566170 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.566291 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.566449 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.566552 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.566621 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260810 22:56:52.566761 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.563265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.0.0.1:40412)
0810 22:56:52.563512 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.563529 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.563620 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.563851 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.563853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.563869 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:52.563963 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.563967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.564743 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.564746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.565515 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.565524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.566297 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.566300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.566428 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.566455 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.566537 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.566624 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260810 22:56:52.567759 5338 raft_consensus.cc:1203] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 22:56:52.568099 5372 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6318f97f3e634a06ab2be1f8cddbcecb. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.568282 5372 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: This master's current role is: LEARNER
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:52.569622 5682 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260810 22:56:52.570477 5682 webserver.cc:460] Webserver started at http://127.29.78.131:45575/ using document root <none> and password file <none>
I20260810 22:56:52.571048 5682 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:52.571144 5682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:52.574893 5682 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 22:56:52.576898 5701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.578248 5682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 22:56:52.578374 5682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "0465065cac1c434a9cec3e07c19a8590"
format_stamp: "Formatted at 2026-08-10 22:56:51 on dist-test-slave-6nmv"
server_key: "6eb18e9168f76b1f3d27e616e09cc2f5"
server_key_iv: "ceca3766dfe81ecc9fa00bad471eb161"
server_key_version: "encryptionkey@0"
I20260810 22:56:52.578902 5682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:52.600045 5682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:52.604205 5682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:52.604408 5682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:52.605104 5682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:56:52.606210 5682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:52.606268 5682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.606329 5682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:52.606360 5682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:52.615535 5682 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:37281
I20260810 22:56:52.615603 5821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:37281 every 8 connection(s)
I20260810 22:56:52.616547 5682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:52.618911 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 5682
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:52.631747 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.621253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40426 (local address 127.29.78.188:44697)
0810 22:56:52.621623 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.621652 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.622392 (+ 740us) server_negotiation.cc:407] Connection header received
0810 22:56:52.622568 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.622572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.622624 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.622684 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.623618 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.624433 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.625539 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.625906 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.625983 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.627852 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.627855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.627857 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.627884 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.629730 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.630260 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.630264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.630268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.630362 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.630707 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.630711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.630713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.630934 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.631052 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.631421 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.631547 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":99,"threads_started":1}
I20260810 22:56:52.633608 4499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.618326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:47827 (local address 127.29.78.190:42665)
0810 22:56:52.618474 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.618478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.620648 (+ 2170us) server_negotiation.cc:407] Connection header received
0810 22:56:52.621804 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.621851 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.621910 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.621989 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.623625 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.624405 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.625983 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.626294 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.626378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.629558 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.629573 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.629575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.629604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.631366 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.631970 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.631973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.631975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.632061 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.632462 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.632465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.632467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.632627 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.632727 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.633314 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.633431 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:56:52.633935 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.621265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.29.78.189:46823)
0810 22:56:52.621468 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.621471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.622045 (+ 574us) server_negotiation.cc:407] Connection header received
0810 22:56:52.622349 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.622355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.622416 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.622478 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.623579 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.624427 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.625885 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.626253 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.626339 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.628696 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.628699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.628702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.628731 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.630906 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.631425 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.631429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.631431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.631524 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.631993 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.631997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.631998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.632240 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.632334 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.633596 (+ 1262us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.633730 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260810 22:56:52.635087 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.618681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:47827)
0810 22:56:52.620480 (+ 1799us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.620535 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.621564 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.622298 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.622306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.622811 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.623460 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.623479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.624580 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.624583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.625848 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.625858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.626592 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.626596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.626811 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.627394 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.627419 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.629341 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.631478 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.631483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.631510 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.631842 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.632194 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.632200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.632205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.632370 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.632775 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.632780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.633064 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.633756 (+ 692us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.634107 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1580,"thread_start_us":659,"threads_started":1}
I20260810 22:56:52.636361 5823 heartbeater.cc:347] Connected to a master server at 127.29.78.190:42665
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.188@KRBTEST.COM
I20260810 22:56:52.636613 5823 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.637197 5823 heartbeater.cc:511] Master 127.29.78.190:42665 requested a full tablet report, sending...
Aug 10 22:56:52 dist-test-slave-6nmv krb5kdc[4408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402612, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.189@KRBTEST.COM
I20260810 22:56:52.638742 4458 ts_manager.cc:194] Re-registered known tserver with Master: 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131:37281)
I20260810 22:56:52.639086 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.621061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56108 (local address 127.29.78.190:42665)
0810 22:56:52.622092 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.622096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.622110 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:52.622427 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.622430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.622482 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.622562 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.624138 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.624820 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.632671 (+ 7851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.632975 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.633065 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.634958 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.634961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.634964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.634985 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.636785 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.637290 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.637295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.637299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.637386 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.637721 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.637727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.637731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.637990 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.638122 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.638726 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.638865 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":66,"threads_started":1}
I20260810 22:56:52.639436 4458 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:47827
I20260810 22:56:52.641047 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.620008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:32819 (local address 127.29.78.188:44697)
0810 22:56:52.620211 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.620216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.622897 (+ 2681us) server_negotiation.cc:407] Connection header received
0810 22:56:52.624981 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.624984 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.625037 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.625092 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.626648 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.627442 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.632142 (+ 4700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.632452 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.632526 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.637029 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.637033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.637036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.637060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.639162 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.639667 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.639670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.639672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.639803 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.640168 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.640173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.640175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.640398 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.640490 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.640790 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.640898 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260810 22:56:52.641470 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.618949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.29.78.131:32819)
0810 22:56:52.622801 (+ 3852us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.622816 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.623326 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.625232 (+ 1906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.625235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.625472 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.625633 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.625640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.627600 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.627604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.628507 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.628517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.632737 (+ 4220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.632740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.632841 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.634899 (+ 2058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.634916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.636832 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.639272 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.639277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.639281 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.639517 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.639938 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.639944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.639948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.640052 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.640534 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.640537 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.640612 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.641168 (+ 556us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.641290 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3786,"thread_start_us":148,"threads_started":1}
I20260810 22:56:52.641685 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.619879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:42255 (local address 127.29.78.189:46823)
0810 22:56:52.620013 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.620021 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.622370 (+ 2349us) server_negotiation.cc:407] Connection header received
0810 22:56:52.625835 (+ 3465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.625839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.625928 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.625985 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:52.628676 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.629455 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.633204 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.633523 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.633596 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.637785 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:52.637791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:52.637796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:52.637818 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:52.639559 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.640209 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.640214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.640215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.640311 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:52.640644 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:52.640648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:52.640650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:52.640892 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:52.640979 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.641341 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.641492 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:56:52.642119 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.618681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.29.78.131:42255)
0810 22:56:52.622280 (+ 3599us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.622296 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.623191 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.626123 (+ 2932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.626126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.626335 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:52.626503 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.626512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.629699 (+ 3187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.629703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.630693 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.630706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.634290 (+ 3584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.634296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.634410 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.635912 (+ 1502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:52.635932 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:52.637619 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:52.639878 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.639881 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.639883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.640097 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.640434 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:52.640437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:52.640439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:52.640534 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:52.641050 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:52.641053 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:52.641145 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.641853 (+ 708us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.641975 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3488,"mutex_wait_us":38,"spinlock_wait_cycles":38784,"thread_start_us":168,"threads_started":1}
I20260810 22:56:52.642452 5822 heartbeater.cc:347] Connected to a master server at 127.29.78.188:44697
I20260810 22:56:52.642539 5822 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.642681 5824 heartbeater.cc:347] Connected to a master server at 127.29.78.189:46823
I20260810 22:56:52.642731 5822 heartbeater.cc:511] Master 127.29.78.188:44697 requested a full tablet report, sending...
I20260810 22:56:52.642761 5824 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:52.642890 5824 heartbeater.cc:511] Master 127.29.78.189:46823 requested a full tablet report, sending...
I20260810 22:56:52.643445 5328 ts_manager.cc:194] Registered new tserver with Master: 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131:37281)
I20260810 22:56:52.643864 5126 ts_manager.cc:194] Registered new tserver with Master: 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131:37281)
I20260810 22:56:52.644398 4458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:52.647884 4458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:52.650427 5294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: NON_VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: true } } }
I20260810 22:56:52.650563 5294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:52.654177 5294 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: attempting to promote NON_VOTER eb2f666902964b2dbaa5fe0bae8ac1bc to VOTER
I20260810 22:56:52.655224 5294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:52.656378 5338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 LEARNER]: Refusing update from remote peer 6318f97f3e634a06ab2be1f8cddbcecb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 22:56:52.656756 5843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 22:56:52.657791 5136 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Refusing update from remote peer 6318f97f3e634a06ab2be1f8cddbcecb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 22:56:52.658196 5843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [LEADER]: Connected to new peer: Peer: permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 22:56:52.658504 5843 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.659075 5338 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.660037 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.660128 5294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.660239 5294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:52.660264 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.660347 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:52.660916 5136 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.661386 5372 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.661494 5372 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:56:52.660135 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:52.663899 5856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:52.664069 5856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:56:52.668802 5857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.662416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:42085 (local address 127.0.0.1:40988)
0810 22:56:52.663357 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.663367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.663435 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.664142 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.664144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.664162 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:52.664284 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.664292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.666009 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.666013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.666922 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.666943 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.668163 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.668166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.668435 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.668460 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.668520 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.668664 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":57,"thread_start_us":109,"threads_started":1}
I20260810 22:56:52.668905 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.663331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.29.78.130:42085)
0810 22:56:52.663648 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.663657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.663675 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:56:52.663758 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.663765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.663977 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.664191 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:52.664381 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.665898 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.667047 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.667748 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.667852 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.668638 (+ 786us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.668688 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.668777 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":110,"threads_started":1}
I20260810 22:56:52.669719 5695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.661993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36833 (local address 127.0.0.1:47128)
0810 22:56:52.662107 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.662122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.662201 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.665543 (+ 3342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.665546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.665562 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:52.665645 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.665650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.667166 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.667170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.668296 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.668306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.669322 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.669326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.669458 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.669479 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.669549 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.669632 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":54}
I20260810 22:56:52.670244 5854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.662368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.29.78.129:36833)
0810 22:56:52.664713 (+ 2345us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.664727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.664749 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:56:52.664833 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.664838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.665057 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.665580 (+ 523us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:52.665819 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.667021 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.668416 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.669210 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.669317 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.669460 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.670002 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.670111 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2232,"thread_start_us":149,"threads_started":1}
I20260810 22:56:52.670776 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.663026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37281 (local address 127.0.0.1:51690)
0810 22:56:52.665298 (+ 2272us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.665312 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.665410 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.666394 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.666397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.666415 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:52.666497 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.666503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.668454 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.668457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.669451 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.669460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.670443 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.670446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.670527 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.670546 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.670592 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.670667 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2208,"mutex_wait_us":18,"thread_start_us":82,"threads_started":1}
I20260810 22:56:52.671435 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.663836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51690 (local address 127.29.78.131:37281)
0810 22:56:52.665939 (+ 2103us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.665951 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.665973 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:56:52.666042 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.666047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.666225 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.666679 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:52.666802 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.668338 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.669550 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.670290 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.671057 (+ 767us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.671182 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.671226 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.671306 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1997,"thread_start_us":107,"threads_started":1}
I20260810 22:56:52.672477 5603 tablet_service.cc:1467] Processing CreateTablet for tablet ada96db8b4f84f028d724447d21a23fd (DEFAULT_TABLE table=test-table [id=1ba87d0d76044854bb3257f877e643a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:52.673786 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada96db8b4f84f028d724447d21a23fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:52.674193 5451 tablet_service.cc:1467] Processing CreateTablet for tablet ada96db8b4f84f028d724447d21a23fd (DEFAULT_TABLE table=test-table [id=1ba87d0d76044854bb3257f877e643a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:52.674193 5752 tablet_service.cc:1467] Processing CreateTablet for tablet ada96db8b4f84f028d724447d21a23fd (DEFAULT_TABLE table=test-table [id=1ba87d0d76044854bb3257f877e643a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:52.675168 5451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada96db8b4f84f028d724447d21a23fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:52.675168 5752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ada96db8b4f84f028d724447d21a23fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:52.685155 5860 tablet_bootstrap.cc:492] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Bootstrap starting.
I20260810 22:56:52.685285 5862 tablet_bootstrap.cc:492] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Bootstrap starting.
I20260810 22:56:52.686122 5861 tablet_bootstrap.cc:492] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Bootstrap starting.
I20260810 22:56:52.688463 5861 tablet_bootstrap.cc:654] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:52.688530 5860 tablet_bootstrap.cc:654] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:52.689406 5861 log.cc:824] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:52.689406 5860 log.cc:824] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:52.691465 5860 tablet_bootstrap.cc:492] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: No bootstrap required, opened a new log
I20260810 22:56:52.691676 5860 ts_tablet_manager.cc:1397] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 22:56:52.692704 5861 tablet_bootstrap.cc:492] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: No bootstrap required, opened a new log
I20260810 22:56:52.692884 5861 ts_tablet_manager.cc:1397] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:56:52.696583 5860 raft_consensus.cc:348] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.696916 5860 raft_consensus.cc:374] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:52.696979 5860 raft_consensus.cc:729] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd6b2a2130d427c8bbe156ed85f7d06, State: Initialized, Role: FOLLOWER
I20260810 22:56:52.697410 5860 consensus_queue.cc:260] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.697604 5861 raft_consensus.cc:348] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.697861 5861 raft_consensus.cc:374] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:52.697921 5861 raft_consensus.cc:729] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0465065cac1c434a9cec3e07c19a8590, State: Initialized, Role: FOLLOWER
I20260810 22:56:52.698762 5860 ts_tablet_manager.cc:1428] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:56:52.698330 5861 consensus_queue.cc:260] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.699838 5861 ts_tablet_manager.cc:1428] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:56:52.700147 5524 heartbeater.cc:503] Master 127.29.78.190:42665 was elected leader, sending a full tablet report...
I20260810 22:56:52.701733 5823 heartbeater.cc:503] Master 127.29.78.190:42665 was elected leader, sending a full tablet report...
I20260810 22:56:52.706674 5862 tablet_bootstrap.cc:654] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:52.707705 5862 log.cc:824] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:52.709975 5862 tablet_bootstrap.cc:492] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: No bootstrap required, opened a new log
I20260810 22:56:52.710167 5862 ts_tablet_manager.cc:1397] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20260810 22:56:52.716125 5862 raft_consensus.cc:348] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.716418 5862 raft_consensus.cc:374] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:52.716483 5862 raft_consensus.cc:729] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f13c2dd9314170b397a16a4a650007, State: Initialized, Role: FOLLOWER
I20260810 22:56:52.717000 5862 consensus_queue.cc:260] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.718137 5862 ts_tablet_manager.cc:1428] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260810 22:56:52.718210 5674 heartbeater.cc:503] Master 127.29.78.190:42665 was elected leader, sending a full tablet report...
W20260810 22:56:52.723457 5526 tablet.cc:2369] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:56:52.868362 5825 tablet.cc:2369] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:56:52.908159 5677 tablet.cc:2369] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:52.944546 5866 raft_consensus.cc:482] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:52.944793 5866 raft_consensus.cc:504] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.946313 5866 leader_election.cc:288] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129:36833), 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085)
I20260810 22:56:52.951340 5858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.946707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:54987 (local address 127.29.78.130:42085)
0810 22:56:52.946845 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.946852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.946893 (+ 41us) server_negotiation.cc:407] Connection header received
0810 22:56:52.946978 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.946982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.947032 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.947111 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:52.947458 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.948229 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.950102 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.950670 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.950784 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.950850 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.951141 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.951213 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:56:52.951340 5854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.946621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:48859 (local address 127.29.78.129:36833)
0810 22:56:52.946903 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:52.946909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:52.946927 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:56:52.946978 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:52.946982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:52.947032 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:52.947111 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:52.947458 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.948230 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.950102 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.950692 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.950787 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.950849 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:52.951141 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 22:56:52.951217 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260810 22:56:52.951364 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.946552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36833 (local address 127.29.78.131:48859)
0810 22:56:52.946759 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.946775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.946878 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.947235 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.947237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.947264 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:52.947343 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.947348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.948350 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.948353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.949968 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.949979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.950788 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.950791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.950927 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.951019 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.951114 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.951204 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":278,"spinlock_wait_cycles":5504}
I20260810 22:56:52.951364 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:52.946620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:42085 (local address 127.29.78.131:54987)
0810 22:56:52.946759 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:52.946775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:52.946885 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:52.947136 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:52.947139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:52.947162 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:52.947332 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.947339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.948350 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.948353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.949968 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:52.949979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:52.950788 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:52.950791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:52.951012 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:52.951028 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 22:56:52.951114 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 22:56:52.951204 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":187,"spinlock_wait_cycles":10752}
I20260810 22:56:52.952083 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada96db8b4f84f028d724447d21a23fd" candidate_uuid: "0465065cac1c434a9cec3e07c19a8590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" is_pre_election: true
I20260810 22:56:52.952174 5626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada96db8b4f84f028d724447d21a23fd" candidate_uuid: "0465065cac1c434a9cec3e07c19a8590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f13c2dd9314170b397a16a4a650007" is_pre_election: true
I20260810 22:56:52.952417 5475 raft_consensus.cc:2454] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0465065cac1c434a9cec3e07c19a8590 in term 0.
I20260810 22:56:52.952481 5626 raft_consensus.cc:2454] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0465065cac1c434a9cec3e07c19a8590 in term 0.
I20260810 22:56:52.952838 5703 leader_election.cc:302] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0465065cac1c434a9cec3e07c19a8590, 5fd6b2a2130d427c8bbe156ed85f7d06; no voters:
I20260810 22:56:52.953090 5866 raft_consensus.cc:2781] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:52.953169 5866 raft_consensus.cc:482] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:52.953215 5866 raft_consensus.cc:3037] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:52.954257 5866 raft_consensus.cc:504] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.954550 5866 leader_election.cc:288] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [CANDIDATE]: Term 1 election: Requested vote from peers 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129:36833), 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085)
I20260810 22:56:52.954816 5475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada96db8b4f84f028d724447d21a23fd" candidate_uuid: "0465065cac1c434a9cec3e07c19a8590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06"
I20260810 22:56:52.954877 5626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ada96db8b4f84f028d724447d21a23fd" candidate_uuid: "0465065cac1c434a9cec3e07c19a8590" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f13c2dd9314170b397a16a4a650007"
I20260810 22:56:52.954974 5626 raft_consensus.cc:3037] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:52.954964 5475 raft_consensus.cc:3037] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:52.956199 5475 raft_consensus.cc:2454] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0465065cac1c434a9cec3e07c19a8590 in term 1.
I20260810 22:56:52.956199 5626 raft_consensus.cc:2454] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0465065cac1c434a9cec3e07c19a8590 in term 1.
I20260810 22:56:52.956490 5703 leader_election.cc:302] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0465065cac1c434a9cec3e07c19a8590, 5fd6b2a2130d427c8bbe156ed85f7d06; no voters:
I20260810 22:56:52.956735 5866 raft_consensus.cc:2781] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:52.956995 5866 raft_consensus.cc:686] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 1 LEADER]: Becoming Leader. State: Replica: 0465065cac1c434a9cec3e07c19a8590, State: Running, Role: LEADER
I20260810 22:56:52.957427 5866 consensus_queue.cc:237] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:52.960953 4458 catalog_manager.cc:5495] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "0465065cac1c434a9cec3e07c19a8590" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } health_report { overall_health: HEALTHY } } }
I20260810 22:56:53.037644 5859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:53.034141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51698 (local address 127.29.78.131:37281)
0810 22:56:53.034273 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:53.034279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:53.034296 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:56:53.034387 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:53.034389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:53.034460 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:53.034567 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:53.034813 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.035313 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.036188 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.036416 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.036525 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:53.036824 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:53.037242 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:53.037374 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:53.037423 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 22:56:53.037476 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:56:53.043284 5626 raft_consensus.cc:1261] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Refusing update from remote peer 0465065cac1c434a9cec3e07c19a8590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:53.043284 5475 raft_consensus.cc:1261] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 1 FOLLOWER]: Refusing update from remote peer 0465065cac1c434a9cec3e07c19a8590: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:53.043951 5866 consensus_queue.cc:1035] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:53.044211 5869 consensus_queue.cc:1035] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:53.060670 5875 mvcc.cc:205] Tried to move back new op lower bound from 7317105103017185280 to 7317105102675464192. Current Snapshot: MvccSnapshot[applied={T|T < 7317105103017185280 or (T in {7317105103017185280})}]
I20260810 22:56:53.063678 4458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:53.063823 4458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:53.071456 4458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6318f97f3e634a06ab2be1f8cddbcecb: Sending DeleteTablet for 3 replicas of tablet ada96db8b4f84f028d724447d21a23fd
I20260810 22:56:53.072973 5451 tablet_service.cc:1514] Processing DeleteTablet for tablet ada96db8b4f84f028d724447d21a23fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:53 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:47128
I20260810 22:56:53.073423 5450 tablet_service.cc:1514] Processing DeleteTablet for tablet ada96db8b4f84f028d724447d21a23fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:53 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:47128
I20260810 22:56:53.073619 5881 tablet_replica.cc:330] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: stopping tablet replica
I20260810 22:56:53.073666 5752 tablet_service.cc:1514] Processing DeleteTablet for tablet ada96db8b4f84f028d724447d21a23fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:53 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:51690
I20260810 22:56:53.074072 5881 raft_consensus.cc:2229] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:53.074463 5603 tablet_service.cc:1514] Processing DeleteTablet for tablet ada96db8b4f84f028d724447d21a23fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:53 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:40988
I20260810 22:56:53.074621 5882 tablet_replica.cc:330] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: stopping tablet replica
I20260810 22:56:53.074890 5882 raft_consensus.cc:2229] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:53.075162 5882 raft_consensus.cc:2258] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:53.076630 5882 ts_tablet_manager.cc:1905] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:53.078810 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 4422
I20260810 22:56:53.079870 5883 tablet_replica.cc:330] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: stopping tablet replica
I20260810 22:56:53.080129 5883 raft_consensus.cc:2229] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:53.080333 5883 raft_consensus.cc:2258] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:53.086107 5882 ts_tablet_manager.cc:1918] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:53.086210 5882 log.cc:1196] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/ada96db8b4f84f028d724447d21a23fd
I20260810 22:56:53.086541 5882 ts_tablet_manager.cc:1939] T ada96db8b4f84f028d724447d21a23fd P 0465065cac1c434a9cec3e07c19a8590: Deleting consensus metadata
W20260810 22:56:53.093050 5704 connection.cc:535] server connection from 127.0.0.1:51690 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 22:56:53.094485 5674 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:42665 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.29.78.190:42665 (error 108)
I20260810 22:56:53.097738 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:53.077007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.29.78.189:46823)
0810 22:56:53.077221 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:53.077225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:53.082223 (+ 4998us) server_negotiation.cc:407] Connection header received
0810 22:56:53.083008 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:53.083011 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:53.083061 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:53.083109 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:53.087783 (+ 4674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.088636 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.089951 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.090334 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.090412 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:53.093131 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:53.093135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:53.093138 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:53.093177 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:53.095334 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.096310 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.096314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.096316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.096402 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.096776 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.096780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.096781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.097002 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:53.097126 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:53.097466 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 22:56:53.097581 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20260810 22:56:53.098358 5881 raft_consensus.cc:2258] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:53.101895 5881 ts_tablet_manager.cc:1905] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:53.102697 5883 ts_tablet_manager.cc:1905] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:53.104611 5881 ts_tablet_manager.cc:1918] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:53.104750 5881 log.cc:1196] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/ada96db8b4f84f028d724447d21a23fd
I20260810 22:56:53.105118 5881 ts_tablet_manager.cc:1939] T ada96db8b4f84f028d724447d21a23fd P 5fd6b2a2130d427c8bbe156ed85f7d06: Deleting consensus metadata
I20260810 22:56:53.105743 5883 ts_tablet_manager.cc:1918] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:53.105897 5883 log.cc:1196] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/ada96db8b4f84f028d724447d21a23fd
I20260810 22:56:53.106262 5883 ts_tablet_manager.cc:1939] T ada96db8b4f84f028d724447d21a23fd P 38f13c2dd9314170b397a16a4a650007: Deleting consensus metadata
W20260810 22:56:53.106451 5401 connection.cc:441] server connection from 127.0.0.1:47128 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47128 (request call id 1) could send its response
W20260810 22:56:53.106917 5401 connection.cc:441] server connection from 127.0.0.1:47128 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47128 (request call id 2) could send its response
W20260810 22:56:53.107404 5550 connection.cc:441] server connection from 127.0.0.1:40988 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40988 (request call id 1) could send its response
I20260810 22:56:53.113924 5528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:53.076148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40436 (local address 127.29.78.188:44697)
0810 22:56:53.076257 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:53.076262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:53.099325 (+ 23063us) server_negotiation.cc:407] Connection header received
0810 22:56:53.099503 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:53.099506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:53.099555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:53.099601 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:53.100382 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.101120 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.102549 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.102846 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.102956 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:53.107844 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:53.107847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:53.107850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:53.107886 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:53.109590 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.112515 (+ 2925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.112519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.112521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.112611 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.112925 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.112930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.112933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.113152 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:53.113264 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:53.113602 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 22:56:53.113715 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:56:53.115235 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:53.105774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.29.78.188:44697)
0810 22:56:53.106020 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:53.106026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:53.106891 (+ 865us) server_negotiation.cc:407] Connection header received
0810 22:56:53.107047 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:53.107062 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:53.107105 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:53.107182 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:53.107864 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.108586 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.109636 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.109975 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.110041 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:53.111527 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:53.111530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:53.111532 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:53.111554 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:53.113157 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.113664 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.113667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.113669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.113745 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.114378 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.114382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.114383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.114575 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:53.114665 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:53.114977 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 22:56:53.115090 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"spinlock_wait_cycles":1920}
I20260810 22:56:53.120975 5259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:53.097977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.29.78.189:46823)
0810 22:56:53.098116 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:53.098120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:53.105439 (+ 7319us) server_negotiation.cc:407] Connection header received
0810 22:56:53.106563 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:53.106566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:53.106614 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:53.106667 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:53.110131 (+ 3464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.111383 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.115278 (+ 3895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:53.115609 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:53.115680 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:53.117753 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:53.117756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:53.117759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:53.117784 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:53.119331 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.119788 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.119792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.119793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.119879 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:53.120145 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:53.120148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:53.120150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:53.120332 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:53.120424 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:53.120715 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 22:56:53.120822 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:56:54.056938 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.056430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:36343)
0810 22:56:54.056717 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.056834 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
W20260810 22:56:54.057286 5823 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:42665 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:54.073118 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.072668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:56349)
0810 22:56:54.072908 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.073017 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":72,"threads_started":1}
W20260810 22:56:54.073442 5524 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:42665 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:54.095522 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.095031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:38991)
0810 22:56:54.095338 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.095425 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":124,"threads_started":1}
I20260810 22:56:54.569591 5896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6318f97f3e634a06ab2be1f8cddbcecb)
I20260810 22:56:54.569900 5896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:54.570434 5897 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6318f97f3e634a06ab2be1f8cddbcecb)
I20260810 22:56:54.570562 5897 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:54.571411 5896 leader_election.cc:288] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665), eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188:44697)
I20260810 22:56:54.571447 5897 leader_election.cc:288] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665), 877d1adeeadf48ee8afa03454a2cc0d0 (127.29.78.189:46823)
I20260810 22:56:54.572184 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.571245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42100)
0810 22:56:54.572011 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.572081 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":147,"thread_start_us":62,"threads_started":1}
I20260810 22:56:54.572271 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.571172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42092)
0810 22:56:54.571740 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.571825 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":73,"threads_started":1}
W20260810 22:56:54.572474 5310 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
W20260810 22:56:54.572517 5111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:54.573342 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.573162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42112)
0810 22:56:54.573243 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.573287 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 22:56:54.573480 5310 leader_election.cc:334] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665): Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:54.573997 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.573774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42118)
0810 22:56:54.573900 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.573945 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 22:56:54.574126 5111 leader_election.cc:334] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665): Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:54.575587 5901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.571436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52518 (local address 127.29.78.188:44697)
0810 22:56:54.571690 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:54.571695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:54.571961 (+ 266us) server_negotiation.cc:407] Connection header received
0810 22:56:54.572155 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:54.572160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:54.572207 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:54.572262 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:54.572678 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.573398 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.574640 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.575062 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.575167 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:54.575242 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:54.575411 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 22:56:54.575477 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
I20260810 22:56:54.575588 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.571240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:44697 (local address 127.0.0.1:52518)
0810 22:56:54.571881 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.571896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:54.572033 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:54.572423 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:54.572425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:54.572442 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:54.572550 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:54.572558 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.573522 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.573525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:54.574532 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:54.574540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.575197 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.575201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:54.575290 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:54.575315 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 22:56:54.575394 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:56:54.575472 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":55,"threads_started":1}
I20260810 22:56:54.576095 5903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.571931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.29.78.189:46823)
0810 22:56:54.572649 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:54.572654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:54.572668 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:54.572713 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:54.572717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:54.572766 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:54.572858 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:54.573146 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.573889 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.575086 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.575505 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.575610 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:54.575810 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:54.575890 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 22:56:54.575972 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":72,"threads_started":1}
I20260810 22:56:54.576097 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.571701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:46823 (local address 127.0.0.1:33656)
0810 22:56:54.572233 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.572249 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:54.572339 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:54.572885 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:54.572887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:54.572903 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:54.573008 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:54.573013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.573980 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.573983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:54.574976 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:54.574985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:54.575622 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:54.575626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:54.575745 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:54.575788 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 22:56:54.575865 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 22:56:54.575965 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":20,"thread_start_us":47,"threads_started":1}
I20260810 22:56:54.576164 5338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" is_pre_election: true
I20260810 22:56:54.576402 5338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 877d1adeeadf48ee8afa03454a2cc0d0 in term 1.
I20260810 22:56:54.576560 5136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" is_pre_election: true
I20260810 22:56:54.576699 5136 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb2f666902964b2dbaa5fe0bae8ac1bc in term 1.
I20260810 22:56:54.576741 5109 leader_election.cc:302] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 877d1adeeadf48ee8afa03454a2cc0d0, eb2f666902964b2dbaa5fe0bae8ac1bc; no voters: 6318f97f3e634a06ab2be1f8cddbcecb
I20260810 22:56:54.576956 5896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:56:54.576997 5307 leader_election.cc:302] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 877d1adeeadf48ee8afa03454a2cc0d0, eb2f666902964b2dbaa5fe0bae8ac1bc; no voters: 6318f97f3e634a06ab2be1f8cddbcecb
I20260810 22:56:54.577026 5896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6318f97f3e634a06ab2be1f8cddbcecb)
I20260810 22:56:54.577070 5896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:56:54.577188 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 22:56:54.577273 5897 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6318f97f3e634a06ab2be1f8cddbcecb)
I20260810 22:56:54.577324 5897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:56:54.578320 5896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:54.578395 5897 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:54.578677 5896 leader_election.cc:288] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 2 election: Requested vote from peers 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665), eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188:44697)
I20260810 22:56:54.578989 5338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
I20260810 22:56:54.579018 5897 leader_election.cc:288] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 election: Requested vote from peers 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665), 877d1adeeadf48ee8afa03454a2cc0d0 (127.29.78.189:46823)
I20260810 22:56:54.579107 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.578854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42128)
0810 22:56:54.578944 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.578992 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 22:56:54.579210 5338 raft_consensus.cc:2379] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 877d1adeeadf48ee8afa03454a2cc0d0 in current term 2: Already voted for candidate eb2f666902964b2dbaa5fe0bae8ac1bc in this term.
I20260810 22:56:54.579272 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.578848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42124)
0810 22:56:54.579072 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.579151 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260810 22:56:54.579679 5136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "877d1adeeadf48ee8afa03454a2cc0d0"
I20260810 22:56:54.579823 5136 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate eb2f666902964b2dbaa5fe0bae8ac1bc in current term 2: Already voted for candidate 877d1adeeadf48ee8afa03454a2cc0d0 in this term.
I20260810 22:56:54.579906 5900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.579676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42148)
0810 22:56:54.579795 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.579839 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 22:56:54.579993 5111 leader_election.cc:334] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665): Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:54.580049 5111 leader_election.cc:302] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 877d1adeeadf48ee8afa03454a2cc0d0; no voters: 6318f97f3e634a06ab2be1f8cddbcecb, eb2f666902964b2dbaa5fe0bae8ac1bc
I20260810 22:56:54.580142 5902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:54.579524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42134)
0810 22:56:54.580029 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:54.580077 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435}
I20260810 22:56:54.580175 5896 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260810 22:56:54.580201 5310 leader_election.cc:334] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665): Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:54.580250 5310 leader_election.cc:302] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb2f666902964b2dbaa5fe0bae8ac1bc; no voters: 6318f97f3e634a06ab2be1f8cddbcecb, 877d1adeeadf48ee8afa03454a2cc0d0
I20260810 22:56:54.580358 5897 raft_consensus.cc:2726] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 22:56:55.058518 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:55.058086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:47237)
0810 22:56:55.058362 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:55.058419 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260810 22:56:55.074384 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:55.074013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:52329)
0810 22:56:55.074246 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:55.074298 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":83,"threads_started":1}
I20260810 22:56:55.096606 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:55.096181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:50413)
0810 22:56:55.096446 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:55.096508 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260810 22:56:56.059763 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.059341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:33443)
0810 22:56:56.059595 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.059655 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":94,"threads_started":1}
I20260810 22:56:56.075487 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.075109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:50997)
0810 22:56:56.075346 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.075401 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":81,"threads_started":1}
I20260810 22:56:56.097652 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.097242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:51077)
0810 22:56:56.097511 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.097564 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20260810 22:56:56.417637 5913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:56.417764 5913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:56.418171 5913 leader_election.cc:288] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665), eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188:44697)
I20260810 22:56:56.418557 5338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" is_pre_election: true
I20260810 22:56:56.418705 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.418340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42370)
0810 22:56:56.418545 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.418596 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260810 22:56:56.418704 5338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 877d1adeeadf48ee8afa03454a2cc0d0 in term 2.
I20260810 22:56:56.419039 5109 leader_election.cc:302] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 877d1adeeadf48ee8afa03454a2cc0d0, eb2f666902964b2dbaa5fe0bae8ac1bc; no voters:
I20260810 22:56:56.419209 5913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 22:56:56.419282 5913 raft_consensus.cc:482] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:56.419351 5913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:56:56.419616 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.419309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42386)
0810 22:56:56.419459 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.419543 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 22:56:56.419734 5111 leader_election.cc:334] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665): Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:56.420653 5913 raft_consensus.cc:504] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:56.420998 5913 leader_election.cc:288] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 3 election: Requested vote from peers 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665), eb2f666902964b2dbaa5fe0bae8ac1bc (127.29.78.188:44697)
I20260810 22:56:56.421293 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.421077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42396)
0810 22:56:56.421188 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.421235 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 22:56:56.421360 5338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc"
I20260810 22:56:56.421479 5338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:56:56.421892 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.421669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42412)
0810 22:56:56.421750 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.421789 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 22:56:56.421977 5111 leader_election.cc:334] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665): Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
I20260810 22:56:56.422463 5338 raft_consensus.cc:2454] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 877d1adeeadf48ee8afa03454a2cc0d0 in term 3.
I20260810 22:56:56.422716 5109 leader_election.cc:302] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 877d1adeeadf48ee8afa03454a2cc0d0, eb2f666902964b2dbaa5fe0bae8ac1bc; no voters: 6318f97f3e634a06ab2be1f8cddbcecb
I20260810 22:56:56.422849 5913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 22:56:56.422973 5913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [term 3 LEADER]: Becoming Leader. State: Replica: 877d1adeeadf48ee8afa03454a2cc0d0, State: Running, Role: LEADER
I20260810 22:56:56.423245 5913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } }
I20260810 22:56:56.424208 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 877d1adeeadf48ee8afa03454a2cc0d0. Latest consensus state: current_term: 3 leader_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:56.424331 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:56.424621 5918 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:56.426911 5918 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:56.427309 5918 catalog_manager.cc:1259] Loaded cluster ID: 2a5f72c4338d4d55b84ee7c2c4923a66
I20260810 22:56:56.427381 5918 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:56.427861 5918 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:56.428203 5918 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Loaded TSK: 0
I20260810 22:56:56.428788 5918 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:56:56.490176 5125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:56.493727 5125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:56.496870 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.496615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42430)
0810 22:56:56.496738 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.496803 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 22:56:56.497007 5338 raft_consensus.cc:1261] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [term 3 FOLLOWER]: Refusing update from remote peer 877d1adeeadf48ee8afa03454a2cc0d0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 22:56:56.497623 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.497323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.0.0.1:42438)
0810 22:56:56.497468 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.497539 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 22:56:56.497542 5916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 22:56:56.497840 5111 consensus_peers.cc:487] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 -> Peer 6318f97f3e634a06ab2be1f8cddbcecb (127.29.78.190:42665): Couldn't send request to peer 6318f97f3e634a06ab2be1f8cddbcecb. Status: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 22:56:56.499001 5920 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 877d1adeeadf48ee8afa03454a2cc0d0. Latest consensus state: current_term: 3 leader_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:56.499155 5920 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:56:56.500113 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:56.500241 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:56.501044 5920 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6318f97f3e634a06ab2be1f8cddbcecb" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 42665 } } peers { permanent_uuid: "877d1adeeadf48ee8afa03454a2cc0d0" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 46823 } attrs { promote: false } } peers { permanent_uuid: "eb2f666902964b2dbaa5fe0bae8ac1bc" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 44697 } attrs { promote: false } } }
I20260810 22:56:56.501154 5920 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb2f666902964b2dbaa5fe0bae8ac1bc [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:56:56.509277 5928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.505880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36833 (local address 127.0.0.1:57192)
0810 22:56:56.506123 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.506138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:56.506239 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:56.506467 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:56.506469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:56.506485 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:56.506601 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.506606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.507351 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.507356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.508169 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.508177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.508883 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.508887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.508980 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.509004 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:56:56.509077 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:56:56.509143 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":17,"thread_start_us":88,"threads_started":1}
I20260810 22:56:56.509298 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.505955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.29.78.129:36833)
0810 22:56:56.506244 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:56.506249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:56.506265 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:56:56.506320 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:56.506323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:56.506365 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:56.506461 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:56.506708 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.507230 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.508279 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.508762 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.508862 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.508922 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:56.509104 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 22:56:56.509178 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":62,"threads_started":1}
I20260810 22:56:56.510015 5450 tablet_service.cc:1467] Processing CreateTablet for tablet 6c83a2d3e5d24195a542187ddbdada12 (DEFAULT_TABLE table=test-table [id=30c68a334e444f0bb0acfb69c3b1aafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:56.510300 5450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c83a2d3e5d24195a542187ddbdada12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:56.510630 5914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.505323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:42085 (local address 127.0.0.1:33558)
0810 22:56:56.505446 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.505456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:56.505532 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:56.507488 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:56.507490 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:56.507505 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:56.507605 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.507609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.508603 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.508605 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.509592 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.509603 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.510281 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.510284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.510361 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.510380 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:56.510461 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 22:56:56.510522 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":3}
I20260810 22:56:56.510667 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.505449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33558 (local address 127.29.78.130:42085)
0810 22:56:56.507238 (+ 1789us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:56.507244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:56.507256 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:56:56.507314 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:56.507317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:56.507374 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:56.507711 (+ 337us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:56.507812 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.508504 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.509725 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.510167 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.510276 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.510330 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:56.510477 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 22:56:56.510558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1710,"thread_start_us":64,"threads_started":1}
I20260810 22:56:56.511116 5603 tablet_service.cc:1467] Processing CreateTablet for tablet 6c83a2d3e5d24195a542187ddbdada12 (DEFAULT_TABLE table=test-table [id=30c68a334e444f0bb0acfb69c3b1aafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:56.511300 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c83a2d3e5d24195a542187ddbdada12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:56.511606 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.506211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37281 (local address 127.0.0.1:36756)
0810 22:56:56.506935 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.506948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:56.507034 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:56.507447 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:56.507450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:56.507466 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:56.507604 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.507610 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.508545 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.508548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.510183 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.510195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.511243 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.511246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.511320 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.511339 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:56.511392 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 22:56:56.511470 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":40,"thread_start_us":88,"threads_started":1}
I20260810 22:56:56.512111 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.506562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.29.78.131:37281)
0810 22:56:56.507199 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:56.507203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:56.507217 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:56.507270 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:56.507272 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:56.507333 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:56.507435 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:56.507714 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.508418 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.510335 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.511120 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.511785 (+ 665us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.511840 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:56.511876 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 22:56:56.511952 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":71,"threads_started":1}
I20260810 22:56:56.513243 5932 tablet_bootstrap.cc:492] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06: Bootstrap starting.
I20260810 22:56:56.513424 5752 tablet_service.cc:1467] Processing CreateTablet for tablet 6c83a2d3e5d24195a542187ddbdada12 (DEFAULT_TABLE table=test-table [id=30c68a334e444f0bb0acfb69c3b1aafc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:56.513659 5752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c83a2d3e5d24195a542187ddbdada12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:56.514143 5932 tablet_bootstrap.cc:654] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:56.515347 5932 tablet_bootstrap.cc:492] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06: No bootstrap required, opened a new log
I20260810 22:56:56.515429 5932 ts_tablet_manager.cc:1397] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:56.515892 5932 raft_consensus.cc:348] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.515995 5932 raft_consensus.cc:374] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:56.516028 5932 raft_consensus.cc:729] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd6b2a2130d427c8bbe156ed85f7d06, State: Initialized, Role: FOLLOWER
I20260810 22:56:56.516237 5936 tablet_bootstrap.cc:492] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: Bootstrap starting.
I20260810 22:56:56.516456 5932 consensus_queue.cc:260] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.516821 5932 ts_tablet_manager.cc:1428] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:56.516969 5934 tablet_bootstrap.cc:492] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: Bootstrap starting.
I20260810 22:56:56.517155 5936 tablet_bootstrap.cc:654] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:56.517561 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.517246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:41327)
0810 22:56:56.517441 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.517494 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260810 22:56:56.518448 5934 tablet_bootstrap.cc:654] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:56.518724 5936 tablet_bootstrap.cc:492] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: No bootstrap required, opened a new log
I20260810 22:56:56.518821 5936 ts_tablet_manager.cc:1397] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:56.519134 5936 raft_consensus.cc:348] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.519253 5936 raft_consensus.cc:374] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:56.519294 5936 raft_consensus.cc:729] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0465065cac1c434a9cec3e07c19a8590, State: Initialized, Role: FOLLOWER
I20260810 22:56:56.519434 5936 consensus_queue.cc:260] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.519608 5934 tablet_bootstrap.cc:492] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: No bootstrap required, opened a new log
I20260810 22:56:56.519685 5934 ts_tablet_manager.cc:1397] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 22:56:56.519804 5936 ts_tablet_manager.cc:1428] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:56.520085 5934 raft_consensus.cc:348] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.520193 5934 raft_consensus.cc:374] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:56.520228 5934 raft_consensus.cc:729] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f13c2dd9314170b397a16a4a650007, State: Initialized, Role: FOLLOWER
I20260810 22:56:56.520342 5934 consensus_queue.cc:260] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.520577 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.520336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:49003)
0810 22:56:56.520471 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.520518 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 22:56:56.520663 5934 ts_tablet_manager.cc:1428] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:56.520908 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.520241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:35763)
0810 22:56:56.520781 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.520832 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260810 22:56:56.521025 5525 heartbeater.cc:503] Master 127.29.78.189:46823 was elected leader, sending a full tablet report...
I20260810 22:56:56.521952 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.521507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:55525)
0810 22:56:56.521775 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.521880 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260810 22:56:56.523430 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.522736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:37223)
0810 22:56:56.523294 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.523360 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478}
I20260810 22:56:56.523865 5824 heartbeater.cc:503] Master 127.29.78.189:46823 was elected leader, sending a full tablet report...
I20260810 22:56:56.524029 5676 heartbeater.cc:503] Master 127.29.78.189:46823 was elected leader, sending a full tablet report...
I20260810 22:56:56.524456 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.524206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:55977)
0810 22:56:56.524327 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.524386 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 22:56:56.754061 5937 raft_consensus.cc:482] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:56.754220 5937 raft_consensus.cc:504] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.755434 5937 leader_election.cc:288] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085), 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131:37281)
I20260810 22:56:56.759328 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.755636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:42085 (local address 127.29.78.129:49623)
0810 22:56:56.755742 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.755757 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:56.755888 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:56.756166 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:56.756169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:56.756215 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:56.756344 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.756349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.757085 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.757090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.758221 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.758231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.758914 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.758917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.759017 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.759048 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 22:56:56.759130 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:56:56.759195 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"spinlock_wait_cycles":3328}
I20260810 22:56:56.759327 5927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.755826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:49623 (local address 127.29.78.130:42085)
0810 22:56:56.755948 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:56.755952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:56.755966 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:56.756018 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:56.756020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:56.756055 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:56.756145 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:56.756461 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.756975 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.758350 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.758750 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.758895 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.758954 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:56.759148 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 22:56:56.759217 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:56.759845 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.755696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:37281 (local address 127.29.78.129:54213)
0810 22:56:56.756045 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.756054 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:56.756206 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:56.756445 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:56.756447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:56.756462 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:56.756589 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.756594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.757305 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.757310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.758620 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:56.758630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.759295 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.759298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:56.759550 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.759582 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 22:56:56.759658 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 22:56:56.759719 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":117,"threads_started":1}
I20260810 22:56:56.759835 5931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.755777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54213 (local address 127.29.78.131:37281)
0810 22:56:56.755896 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:56.755900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:56.756158 (+ 258us) server_negotiation.cc:407] Connection header received
0810 22:56:56.756300 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:56.756303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:56.756352 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:56.756435 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:56.756703 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.757197 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.758749 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.759183 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.759304 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.759356 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:56.759674 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 22:56:56.759745 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:56.759871 5626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c83a2d3e5d24195a542187ddbdada12" candidate_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f13c2dd9314170b397a16a4a650007" is_pre_election: true
I20260810 22:56:56.760037 5626 raft_consensus.cc:2454] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd6b2a2130d427c8bbe156ed85f7d06 in term 0.
I20260810 22:56:56.760399 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c83a2d3e5d24195a542187ddbdada12" candidate_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0465065cac1c434a9cec3e07c19a8590" is_pre_election: true
I20260810 22:56:56.760411 5399 leader_election.cc:302] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f13c2dd9314170b397a16a4a650007, 5fd6b2a2130d427c8bbe156ed85f7d06; no voters:
I20260810 22:56:56.760620 5774 raft_consensus.cc:2454] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd6b2a2130d427c8bbe156ed85f7d06 in term 0.
I20260810 22:56:56.760660 5937 raft_consensus.cc:2781] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:56.760707 5937 raft_consensus.cc:482] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:56.760737 5937 raft_consensus.cc:3037] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:56.761679 5937 raft_consensus.cc:504] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.762042 5937 leader_election.cc:288] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [CANDIDATE]: Term 1 election: Requested vote from peers 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085), 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131:37281)
I20260810 22:56:56.762344 5626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c83a2d3e5d24195a542187ddbdada12" candidate_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f13c2dd9314170b397a16a4a650007"
I20260810 22:56:56.762351 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c83a2d3e5d24195a542187ddbdada12" candidate_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0465065cac1c434a9cec3e07c19a8590"
I20260810 22:56:56.762444 5626 raft_consensus.cc:3037] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:56.762475 5774 raft_consensus.cc:3037] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:56.763334 5626 raft_consensus.cc:2454] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd6b2a2130d427c8bbe156ed85f7d06 in term 1.
I20260810 22:56:56.763535 5774 raft_consensus.cc:2454] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd6b2a2130d427c8bbe156ed85f7d06 in term 1.
I20260810 22:56:56.763650 5399 leader_election.cc:302] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f13c2dd9314170b397a16a4a650007, 5fd6b2a2130d427c8bbe156ed85f7d06; no voters:
I20260810 22:56:56.763875 5937 raft_consensus.cc:2781] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:56.764183 5937 raft_consensus.cc:686] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [term 1 LEADER]: Becoming Leader. State: Replica: 5fd6b2a2130d427c8bbe156ed85f7d06, State: Running, Role: LEADER
I20260810 22:56:56.764473 5937 consensus_queue.cc:237] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } }
I20260810 22:56:56.764999 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.764799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:33539)
0810 22:56:56.764896 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.764943 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 22:56:56.767083 5123 catalog_manager.cc:5495] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fd6b2a2130d427c8bbe156ed85f7d06" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:56.800047 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.796997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.29.78.129:36833)
0810 22:56:56.797125 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:56.797129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:56.797142 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:56:56.797230 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:56.797232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:56.797285 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:56.797364 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:56.797588 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.798123 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.798848 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:56.799029 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:56.799092 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:56.799343 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:56.799719 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:56.799795 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:56.799858 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 22:56:56.799898 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:56:56.802597 5626 raft_consensus.cc:1261] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Refusing update from remote peer 5fd6b2a2130d427c8bbe156ed85f7d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:56.802865 5774 raft_consensus.cc:1261] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 1 FOLLOWER]: Refusing update from remote peer 5fd6b2a2130d427c8bbe156ed85f7d06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:56.803133 5943 consensus_queue.cc:1035] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f13c2dd9314170b397a16a4a650007" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:56.803391 5937 consensus_queue.cc:1035] T 6c83a2d3e5d24195a542187ddbdada12 P 5fd6b2a2130d427c8bbe156ed85f7d06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0465065cac1c434a9cec3e07c19a8590" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 37281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:56.804695 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.804409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:49567)
0810 22:56:56.804559 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.804630 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 22:56:56.805639 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.805365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:57429)
0810 22:56:56.805516 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.805573 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 22:56:56.807123 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.806724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:53539)
0810 22:56:56.806987 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.807044 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260810 22:56:56.808490 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.808218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.130:42843)
0810 22:56:56.808385 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.808434 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 22:56:56.810170 5942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.809964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.129:44181)
0810 22:56:56.810060 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.810111 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 22:56:56.812688 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:56.812456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:42665 (local address 127.29.78.131:39981)
0810 22:56:56.812578 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.812629 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:42665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 22:56:56.814874 5123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:56.815023 5123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:56.818686 5123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 877d1adeeadf48ee8afa03454a2cc0d0: Sending DeleteTablet for 3 replicas of tablet 6c83a2d3e5d24195a542187ddbdada12
I20260810 22:56:56.819288 5603 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c83a2d3e5d24195a542187ddbdada12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:56 UTC) from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:33558
I20260810 22:56:56.819638 5955 tablet_replica.cc:330] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: stopping tablet replica
I20260810 22:56:56.819762 5955 raft_consensus.cc:2229] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:56.819832 5450 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c83a2d3e5d24195a542187ddbdada12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:56 UTC) from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:57192
I20260810 22:56:56.819891 5955 raft_consensus.cc:2258] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:56.819895 5752 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c83a2d3e5d24195a542187ddbdada12 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:56 UTC) from {username='oryx', principal='oryx/127.29.78.189@KRBTEST.COM'} at 127.0.0.1:36756
I20260810 22:56:56.820629 5955 ts_tablet_manager.cc:1905] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:56.820719 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 5282
I20260810 22:56:56.823987 5955 ts_tablet_manager.cc:1918] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:56.824074 5955 log.cc:1196] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/6c83a2d3e5d24195a542187ddbdada12
I20260810 22:56:56.824405 5955 ts_tablet_manager.cc:1939] T 6c83a2d3e5d24195a542187ddbdada12 P 38f13c2dd9314170b397a16a4a650007: Deleting consensus metadata
I20260810 22:56:56.825139 5108 catalog_manager.cc:4841] TS 38f13c2dd9314170b397a16a4a650007 (127.29.78.130:42085): tablet 6c83a2d3e5d24195a542187ddbdada12 (table test-table [id=30c68a334e444f0bb0acfb69c3b1aafc]) successfully deleted
I20260810 22:56:56.830369 5957 tablet_replica.cc:330] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: stopping tablet replica
I20260810 22:56:56.830523 5957 raft_consensus.cc:2229] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:56.830674 5957 raft_consensus.cc:2258] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:56.831328 5957 ts_tablet_manager.cc:1905] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:56:56.832728 5930 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:56.832417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36833 (local address 127.0.0.1:57208)
0810 22:56:56.832546 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:56.832606 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:36833: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 22:56:56.832849 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 5533
W20260810 22:56:56.832927 5109 catalog_manager.cc:4584] TS 5fd6b2a2130d427c8bbe156ed85f7d06 (127.29.78.129:36833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c83a2d3e5d24195a542187ddbdada12: Network error: Client connection negotiation failed: client connection to 127.29.78.129:36833: connect: Connection reset by peer (error 104)
I20260810 22:56:56.837785 5957 ts_tablet_manager.cc:1918] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:56.837886 5957 log.cc:1196] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/6c83a2d3e5d24195a542187ddbdada12
I20260810 22:56:56.838195 5957 ts_tablet_manager.cc:1939] T 6c83a2d3e5d24195a542187ddbdada12 P 0465065cac1c434a9cec3e07c19a8590: Deleting consensus metadata
I20260810 22:56:56.838960 5109 catalog_manager.cc:4841] TS 0465065cac1c434a9cec3e07c19a8590 (127.29.78.131:37281): tablet 6c83a2d3e5d24195a542187ddbdada12 (table test-table [id=30c68a334e444f0bb0acfb69c3b1aafc]) successfully deleted
I20260810 22:56:56.841212 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 5682
I20260810 22:56:56.849562 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 4979
I20260810 22:56:56.856101 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 5095
2026-08-10T22:56:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7491 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:56:56 dist-test-slave-6nmv krb5kdc[5960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:56:56 dist-test-slave-6nmv krb5kdc[5960](info): set up 2 sockets
Aug 10 22:56:56 dist-test-slave-6nmv krb5kdc[5960](info): commencing operation
krb5kdc: starting...
W20260810 22:56:57.899349 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:56:57 dist-test-slave-6nmv krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:56:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:56:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
Entry for principal oryx/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab.
I20260810 22:56:58.028014 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36743
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:58.136394 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:58.136643 5975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:58.136686 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:58.140491 5975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:56:58.140548 5975 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 22:56:58.140568 5975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:58.140589 5975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:58.140607 5975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:56:58.140625 5975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:56:58.145272 5975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36743
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.190.keytab
--principal=oryx/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:58.146193 5975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:58.147120 5975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:58.156129 5982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:58.156308 5984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:58.156236 5975 server_base.cc:1034] running on GCE node
W20260810 22:56:58.156517 5981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:58.157078 5975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:58.158257 5975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:58.159432 5975 hybrid_clock.cc:648] HybridClock initialized: now 1786402618159407 us; error 47 us; skew 500 ppm
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:58.162241 5975 init.cc:377] Logged in from keytab as oryx/127.29.78.190@KRBTEST.COM (short username oryx)
I20260810 22:56:58.163065 5975 webserver.cc:460] Webserver started at http://127.29.78.190:37899/ using document root <none> and password file <none>
I20260810 22:56:58.163599 5975 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:58.163658 5975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:58.163805 5975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:58.165604 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "8ea14fb9c37c4696b768b5af79bfb253"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "483553510eee180b79234dcfaaa39e11"
server_key_iv: "b111f171c05633d885424c59376f1a11"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.166143 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "8ea14fb9c37c4696b768b5af79bfb253"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "483553510eee180b79234dcfaaa39e11"
server_key_iv: "b111f171c05633d885424c59376f1a11"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.169437 5975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:56:58.171675 5991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:58.172804 5975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:58.172936 5975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "8ea14fb9c37c4696b768b5af79bfb253"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "483553510eee180b79234dcfaaa39e11"
server_key_iv: "b111f171c05633d885424c59376f1a11"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.173027 5975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:58.186100 5975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:58.188668 5975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:58.188836 5975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:58.196646 5975 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36743
I20260810 22:56:58.196678 6055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36743 every 8 connection(s)
I20260810 22:56:58.197551 5975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:56:58.200007 6056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:58.203866 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 5975
I20260810 22:56:58.204012 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:56:58.204316 30010 external_mini_cluster.cc:1428] Setting key 621f797b24c43221530967e58089b43b
I20260810 22:56:58.205708 6056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Bootstrap starting.
I20260810 22:56:58.208349 6056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:58.209206 6056 log.cc:824] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:58.211058 6056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: No bootstrap required, opened a new log
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:58.214989 6056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea14fb9c37c4696b768b5af79bfb253" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36743 } }
I20260810 22:56:58.215288 6056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:58.215361 6056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ea14fb9c37c4696b768b5af79bfb253, State: Initialized, Role: FOLLOWER
I20260810 22:56:58.215885 6056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea14fb9c37c4696b768b5af79bfb253" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36743 } }
I20260810 22:56:58.216051 6056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:56:58.216126 6056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:56:58.216234 6056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:58.217336 6056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea14fb9c37c4696b768b5af79bfb253" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36743 } }
I20260810 22:56:58.217713 6056 leader_election.cc:302] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ea14fb9c37c4696b768b5af79bfb253; no voters:
I20260810 22:56:58.218070 6056 leader_election.cc:288] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:56:58.218185 6062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:58.218472 6062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [term 1 LEADER]: Becoming Leader. State: Replica: 8ea14fb9c37c4696b768b5af79bfb253, State: Running, Role: LEADER
I20260810 22:56:58.218830 6062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea14fb9c37c4696b768b5af79bfb253" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36743 } }
I20260810 22:56:58.219343 6056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:56:58.219501 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.205790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.29.78.190:36743)
0810 22:56:58.206279 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.206298 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.206348 (+ 50us) server_negotiation.cc:407] Connection header received
0810 22:56:58.207189 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.207212 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.207596 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.207914 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.208623 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.209920 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.210724 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.211346 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.211495 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.214087 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.214093 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.214108 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.214144 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.216141 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.216686 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.216691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.216696 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.216820 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.217165 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.217168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.217169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.217698 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.217921 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.218275 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.218604 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":111,"threads_started":1}
I20260810 22:56:58.219926 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ea14fb9c37c4696b768b5af79bfb253. Latest consensus state: current_term: 1 leader_uuid: "8ea14fb9c37c4696b768b5af79bfb253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea14fb9c37c4696b768b5af79bfb253" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36743 } } }
I20260810 22:56:58.220019 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:58.221625 6069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:56:58.221899 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ea14fb9c37c4696b768b5af79bfb253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ea14fb9c37c4696b768b5af79bfb253" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36743 } } }
I20260810 22:56:58.222172 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253 [sys.catalog]: This master's current role is: LEADER
I20260810 22:56:58.224431 6069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:56:58.229164 6069 catalog_manager.cc:1347] Generated new cluster ID: 7d274fbdd41e4cbb9ec17d8ef264c8d8
I20260810 22:56:58.229223 6069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:56:58.249208 6069 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:56:58.250291 6069 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:56:58.257755 6069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Generated new TSK 0
I20260810 22:56:58.258494 6069 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
I20260810 22:56:58.298605 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36743
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:58.404157 6086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:58.404363 6086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:58.404402 6086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:58.408059 6086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:58.408102 6086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:58.408157 6086 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:58.408211 6086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:56:58.414456 6086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:58.415396 6086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:58.416280 6086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:58.425024 6094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:58.425022 6092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:58.425024 6091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:58.425238 6086 server_base.cc:1034] running on GCE node
I20260810 22:56:58.425685 6086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:58.426316 6086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:58.427520 6086 hybrid_clock.cc:648] HybridClock initialized: now 1786402618427503 us; error 35 us; skew 500 ppm
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:58.430146 6086 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260810 22:56:58.430953 6086 webserver.cc:460] Webserver started at http://127.29.78.129:43329/ using document root <none> and password file <none>
I20260810 22:56:58.431483 6086 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:58.431547 6086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:58.431789 6086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:58.433723 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "96656987869c403a84c841a55b052f41"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "e0c8bc7414fc0ef48a3fc1234da3df6d"
server_key_iv: "5d5b0e9943bfea2fb425e36ab81691dc"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.434363 6086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "96656987869c403a84c841a55b052f41"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "e0c8bc7414fc0ef48a3fc1234da3df6d"
server_key_iv: "5d5b0e9943bfea2fb425e36ab81691dc"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.437705 6086 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:56:58.440083 6102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:58.441282 6086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:58.441392 6086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "96656987869c403a84c841a55b052f41"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "e0c8bc7414fc0ef48a3fc1234da3df6d"
server_key_iv: "5d5b0e9943bfea2fb425e36ab81691dc"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.441490 6086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:58.459959 6086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:58.463114 6086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:58.463296 6086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:58.464747 6086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:58.464800 6086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:58.464867 6086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:58.464898 6086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:58.475767 6086 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:46563
I20260810 22:56:58.475837 6229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:46563 every 8 connection(s)
I20260810 22:56:58.476802 6086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:56:58.479336 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.465349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:44703 (local address 127.29.78.190:36743)
0810 22:56:58.465472 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.465475 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.466446 (+ 971us) server_negotiation.cc:407] Connection header received
0810 22:56:58.467437 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.467441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.467522 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.467599 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.469254 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.470051 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.471367 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.471624 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.471759 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.475593 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.475596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.475599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.475633 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.477288 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.477875 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.477878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.477880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.477969 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.478288 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.478291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.478293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.478474 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.478603 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.479060 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.479178 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 22:56:58.479993 6112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.465848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36743 (local address 127.29.78.129:44703)
0810 22:56:58.466282 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.466341 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.467230 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.467881 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.467888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.468398 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:58.469078 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.469097 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.470194 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.470197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.471235 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.471245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.471820 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.471825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.471991 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.472579 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:58.472597 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:58.475394 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:58.477419 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.477426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.477443 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.477762 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.478056 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.478059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.478061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.478192 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.478604 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:58.478610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:58.478838 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.479007 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.479281 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260810 22:56:58.481098 6230 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36743
I20260810 22:56:58.481244 6230 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:58.481689 6230 heartbeater.cc:511] Master 127.29.78.190:36743 requested a full tablet report, sending...
I20260810 22:56:58.483063 6007 ts_manager.cc:194] Registered new tserver with Master: 96656987869c403a84c841a55b052f41 (127.29.78.129:46563)
I20260810 22:56:58.484254 6007 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:44703
I20260810 22:56:58.484799 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 6086
I20260810 22:56:58.484939 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:56:58.485313 30010 external_mini_cluster.cc:1428] Setting key cae2965e3ed624dea015eb096789f547
I20260810 22:56:58.491900 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.483254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:34869 (local address 127.29.78.190:36743)
0810 22:56:58.484934 (+ 1680us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.484938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.484956 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:56:58.484989 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.484992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.485031 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.485112 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.485588 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.486319 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.487406 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.487619 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.487694 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.488969 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.488971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.488973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.488992 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.490292 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.490706 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.490709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.490711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.490783 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.491043 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.491047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.491066 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.491223 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.491328 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.491637 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.491760 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589}
I20260810 22:56:58.491914 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.483133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36743 (local address 127.29.78.129:34869)
0810 22:56:58.483438 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.483453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.483542 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.485145 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.485150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.485325 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:58.485471 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.485478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.486474 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.486478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.487283 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.487297 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.487732 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.487738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.487872 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.488345 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:58.488380 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:58.488836 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:58.490413 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.490417 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.490418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.490613 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.490869 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.490873 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.490875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.490956 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.491319 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:58.491322 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:58.491415 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.491557 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.491742 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
I20260810 22:56:58.513103 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36743
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:58.620465 6240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:58.620708 6240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:58.620787 6240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:58.625541 6240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:58.625619 6240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:58.625682 6240 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:58.625737 6240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:56:58.634339 6240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:58.635535 6240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:58.636920 6240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:58.648396 6245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:58.648451 6246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:58.648535 6249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:58.648797 6240 server_base.cc:1034] running on GCE node
I20260810 22:56:58.649303 6240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:58.650377 6240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:58.651583 6240 hybrid_clock.cc:648] HybridClock initialized: now 1786402618651554 us; error 39 us; skew 500 ppm
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:58.655124 6240 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260810 22:56:58.655961 6240 webserver.cc:460] Webserver started at http://127.29.78.130:43353/ using document root <none> and password file <none>
I20260810 22:56:58.656689 6240 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:58.656773 6240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:58.656975 6240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:58.660079 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "741ea328e0e049e49ac12f595b737760"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "0f05002416a622050531d0e65a8bd7b4"
server_key_iv: "026b5b68f668a2806956df6ddc05db22"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.660913 6240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "741ea328e0e049e49ac12f595b737760"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "0f05002416a622050531d0e65a8bd7b4"
server_key_iv: "026b5b68f668a2806956df6ddc05db22"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.664785 6240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:56:58.667717 6256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:58.669122 6240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:56:58.669253 6240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "741ea328e0e049e49ac12f595b737760"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "0f05002416a622050531d0e65a8bd7b4"
server_key_iv: "026b5b68f668a2806956df6ddc05db22"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.669359 6240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:58.681375 6240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:58.684089 6240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:58.684266 6240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:58.685575 6240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:58.685643 6240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:58.685714 6240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:58.685765 6240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:58.697563 6240 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:37721
I20260810 22:56:58.697631 6384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:37721 every 8 connection(s)
I20260810 22:56:58.698446 6240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:56:58.698701 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 6240
I20260810 22:56:58.698832 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:56:58.699168 30010 external_mini_cluster.cc:1428] Setting key 252f2a0e3c8c082f2f1bfacc70a1fd9e
I20260810 22:56:58.699983 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.687939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:45003 (local address 127.29.78.190:36743)
0810 22:56:58.688070 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.688073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.688088 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:56:58.689037 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.689040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.689120 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.689219 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.690829 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.691362 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.692577 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.692822 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.692943 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.695933 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.695939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.695945 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.695978 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.697746 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.698279 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.698283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.698285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.698373 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.698767 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.698772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.698776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.699024 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.699133 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.699732 (+ 599us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.699846 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:56:58.700733 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.686824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36743 (local address 127.29.78.130:45003)
0810 22:56:58.687381 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.687758 (+ 377us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.688796 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.689404 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.689414 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.689953 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:58.690654 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.690672 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.691526 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.691529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.692440 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.692450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.692983 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.692987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.693141 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.693689 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:58.693708 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:58.695766 (+ 2058us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:58.697913 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.697918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.697930 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.698178 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.698483 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.698485 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.698487 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.698654 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.699278 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:58.699283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:58.699522 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.699680 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.699953 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":130,"threads_started":1}
I20260810 22:56:58.701892 6385 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36743
I20260810 22:56:58.702018 6385 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:58.702548 6385 heartbeater.cc:511] Master 127.29.78.190:36743 requested a full tablet report, sending...
I20260810 22:56:58.703447 6008 ts_manager.cc:194] Registered new tserver with Master: 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721)
I20260810 22:56:58.704183 6008 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:45003
WARNING: no policy specified for oryx/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:56:58.710983 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.703988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:38601 (local address 127.29.78.190:36743)
0810 22:56:58.704102 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.704106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.704310 (+ 204us) server_negotiation.cc:407] Connection header received
0810 22:56:58.704462 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.704465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.704513 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.704559 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.705144 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.705915 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.706664 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.706938 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.707049 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.708171 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.708174 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.708177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.708199 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.709524 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.709931 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.709933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.709935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.710016 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.710252 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.710256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.710258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.710443 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.710542 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.710768 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.710861 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:56:58.710992 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.703951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36743 (local address 127.29.78.130:38601)
0810 22:56:58.704227 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.704237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.704367 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.704678 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.704681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.704865 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:58.705004 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.705011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.706034 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.706036 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.706564 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.706572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.707055 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.707059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.707157 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.707664 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:58.707674 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:58.708040 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:58.709626 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.709630 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.709632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.709852 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.710088 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.710091 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.710093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.710171 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.710533 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:58.710536 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:58.710617 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.710719 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.710841 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
I20260810 22:56:58.728176 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36743
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:56:58.834069 6394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:56:58.834302 6394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:56:58.834347 6394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:56:58.838245 6394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:56:58.838292 6394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:56:58.838361 6394 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 22:56:58.838457 6394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:56:58.843354 6394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.29.78.190:36743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:56:58.844442 6394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:56:58.845556 6394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:56:58.855132 6399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:58.855118 6400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:56:58.855108 6402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:56:58.855170 6394 server_base.cc:1034] running on GCE node
I20260810 22:56:58.855748 6394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:56:58.856410 6394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:56:58.857599 6394 hybrid_clock.cc:648] HybridClock initialized: now 1786402618857568 us; error 43 us; skew 500 ppm
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:56:58.860530 6394 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260810 22:56:58.861315 6394 webserver.cc:460] Webserver started at http://127.29.78.131:40731/ using document root <none> and password file <none>
I20260810 22:56:58.861883 6394 fs_manager.cc:359] Metadata directory not provided
I20260810 22:56:58.861953 6394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:56:58.862145 6394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:56:58.864303 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "4f693636c83d4d6ebf9f2d7f5c897532"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "78701fd5e878e06d8445781b1c03bf99"
server_key_iv: "6cbc1b00c0d2e3aa5f3b7e0b0ce7aa1e"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.864961 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "4f693636c83d4d6ebf9f2d7f5c897532"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "78701fd5e878e06d8445781b1c03bf99"
server_key_iv: "6cbc1b00c0d2e3aa5f3b7e0b0ce7aa1e"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.868721 6394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:56:58.871119 6410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:58.872330 6394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:58.872478 6394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "4f693636c83d4d6ebf9f2d7f5c897532"
format_stamp: "Formatted at 2026-08-10 22:56:58 on dist-test-slave-6nmv"
server_key: "78701fd5e878e06d8445781b1c03bf99"
server_key_iv: "6cbc1b00c0d2e3aa5f3b7e0b0ce7aa1e"
server_key_version: "encryptionkey@0"
I20260810 22:56:58.872583 6394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:56:58.891152 6394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:56:58.894465 6394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:56:58.894659 6394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:56:58.896013 6394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:56:58.896082 6394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:56:58.896152 6394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:56:58.896200 6394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:58.906315 6394 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:42511
I20260810 22:56:58.906373 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:42511 every 8 connection(s)
I20260810 22:56:58.907207 6394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:56:58.910404 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.896933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:39409 (local address 127.29.78.190:36743)
0810 22:56:58.897111 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.897117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.897971 (+ 854us) server_negotiation.cc:407] Connection header received
0810 22:56:58.899098 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.899103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.899167 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.899266 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.900927 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.901800 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.903696 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.903919 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.903993 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.906927 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.906931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.906934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.906967 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.908484 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.908975 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.908983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.908984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.909053 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.909362 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.909367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.909371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.909538 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.909651 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.910135 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.910246 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:56:58.911101 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.897311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36743 (local address 127.29.78.131:39409)
0810 22:56:58.897759 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.897825 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.898852 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.899586 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.899596 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.900037 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:58.900751 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.900772 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.901984 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.901988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.903569 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.903581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.904070 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.904074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.904296 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.904951 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:58.904972 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:58.906756 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:58.908592 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.908597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.908610 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.908876 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.909136 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.909140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.909142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.909271 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.909642 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:58.909648 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:58.909925 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.910094 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.910378 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":128,"threads_started":1}
I20260810 22:56:58.912199 6536 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36743
I20260810 22:56:58.912330 6536 heartbeater.cc:464] Registering TS with master...
I20260810 22:56:58.912882 6536 heartbeater.cc:511] Master 127.29.78.190:36743 requested a full tablet report, sending...
I20260810 22:56:58.913982 6008 ts_manager.cc:194] Registered new tserver with Master: 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511)
I20260810 22:56:58.913983 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 6394
I20260810 22:56:58.914112 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:56:58.914400 30010 external_mini_cluster.cc:1428] Setting key 525a35ffc252ca47ae6f5231362995b3
I20260810 22:56:58.914664 6008 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:39409
I20260810 22:56:58.916576 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:56:58.922254 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.914465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:56089 (local address 127.29.78.190:36743)
0810 22:56:58.914590 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.914594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.914919 (+ 325us) server_negotiation.cc:407] Connection header received
0810 22:56:58.915174 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.915179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.915218 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.915280 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.915933 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.916640 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.917777 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.918019 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.918093 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.919270 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.919273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.919275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.919296 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.920664 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.921149 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.921151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.921152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.921220 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.921521 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.921523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.921524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.921652 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.921750 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.922018 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.922117 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 22:56:58.922266 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.914397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36743 (local address 127.29.78.131:56089)
0810 22:56:58.914839 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.914863 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.915083 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.915381 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.915400 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.915656 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:58.915812 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.915821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.916778 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.916782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.917663 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.917676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.918127 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.918132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.918245 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.918674 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:58.918692 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:58.919161 (+ 469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:58.920776 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.920781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.920785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.921066 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.921305 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:58.921310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:58.921315 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:58.921426 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:58.921744 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:58.921747 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:58.921853 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.921965 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.922100 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":187,"threads_started":1}
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:56:58.942370 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:58.936237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.29.78.190:36743)
0810 22:56:58.936390 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.936394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.936407 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:56:58.936446 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.936448 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.936487 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.936562 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.937280 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.937782 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.938639 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.938835 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.938909 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.942232 (+ 3323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45336: BlockingRecv error: recv got EOF from 127.0.0.1:45336 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 22:56:58 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:58.951325 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.943813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45338 (local address 127.29.78.190:36743)
0810 22:56:58.943936 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.943939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.944079 (+ 140us) server_negotiation.cc:407] Connection header received
0810 22:56:58.944151 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.944153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.944188 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.944249 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:58.944828 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.945314 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.946158 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.946384 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.946463 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.948580 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:58.948582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:58.948584 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:58.948604 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:58.950079 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.950438 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.950440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.950441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.950507 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:58.950701 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:58.950702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:58.950703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:58.950842 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:58.950929 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.951065 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.951141 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:56:58.954234 6008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:58.957049 6008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:58.970973 6563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.965209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42511 (local address 127.0.0.1:35516)
0810 22:56:58.966102 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.966115 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.966201 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.966820 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.966823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.966837 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:58.966961 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.966968 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.968328 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.968331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.969276 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.969286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.970376 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.970380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.970487 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.970526 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.970658 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.970783 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260810 22:56:58.971011 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.965603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.29.78.131:42511)
0810 22:56:58.966011 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.966020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.966312 (+ 292us) server_negotiation.cc:407] Connection header received
0810 22:56:58.966394 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.966399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.966542 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.966682 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:58.967091 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.968215 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.969411 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.970241 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.970360 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.970536 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.970711 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.970804 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":99,"threads_started":1}
I20260810 22:56:58.971176 6562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.964821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37721 (local address 127.0.0.1:37978)
0810 22:56:58.965552 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.965605 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.965724 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.966692 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.966695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.966711 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:58.966865 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.966874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.968075 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.968078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.969448 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.969459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.970738 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.970741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.970840 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.970862 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.970952 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.971040 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":5,"thread_start_us":99,"threads_started":1}
I20260810 22:56:58.971495 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.965594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48396 (local address 127.29.78.129:46563)
0810 22:56:58.965973 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.965990 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.966138 (+ 148us) server_negotiation.cc:407] Connection header received
0810 22:56:58.966275 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.966279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.966450 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.966613 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:58.966919 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.968144 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.969633 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.970262 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.970384 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.970558 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.971040 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.971364 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":66,"threads_started":1}
I20260810 22:56:58.970974 6566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.965010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46563 (local address 127.0.0.1:48396)
0810 22:56:58.966023 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:58.966035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:58.966148 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:58.966601 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:58.966607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:58.966629 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:58.966797 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.966804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.968286 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.968289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.969537 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:58.969548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.970635 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.970640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:58.970725 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.970744 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 22:56:58.970818 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 22:56:58.970887 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":98,"thread_start_us":87,"threads_started":1}
I20260810 22:56:58.971719 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:58.964977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.29.78.130:37721)
0810 22:56:58.966179 (+ 1202us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:58.966189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:58.966209 (+ 20us) server_negotiation.cc:407] Connection header received
0810 22:56:58.966304 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:58.966309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:58.966529 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:58.967014 (+ 485us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:58.967151 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.967854 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.969585 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:58.970398 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:58.970494 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:58.970673 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:58.971456 (+ 783us) server_negotiation.cc:299] Negotiation successful
0810 22:56:58.971550 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":103,"threads_started":1}
I20260810 22:56:58.974282 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 2970b332b25c434cb650d83d19bc7fd5 (DEFAULT_TABLE table=test-table [id=a82c7a5efde2437a9e6eb7f8f2354d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:58.974283 6468 tablet_service.cc:1467] Processing CreateTablet for tablet 2970b332b25c434cb650d83d19bc7fd5 (DEFAULT_TABLE table=test-table [id=a82c7a5efde2437a9e6eb7f8f2354d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:58.975396 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2970b332b25c434cb650d83d19bc7fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:58.975404 6468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2970b332b25c434cb650d83d19bc7fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:58.979961 6158 tablet_service.cc:1467] Processing CreateTablet for tablet 2970b332b25c434cb650d83d19bc7fd5 (DEFAULT_TABLE table=test-table [id=a82c7a5efde2437a9e6eb7f8f2354d00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:58.981096 6158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2970b332b25c434cb650d83d19bc7fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:58.983515 6570 tablet_bootstrap.cc:492] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Bootstrap starting.
I20260810 22:56:58.984854 6571 tablet_bootstrap.cc:492] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Bootstrap starting.
I20260810 22:56:58.985934 6570 tablet_bootstrap.cc:654] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:58.987023 6571 tablet_bootstrap.cc:654] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:58.987314 6570 log.cc:824] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:58.987856 6571 log.cc:824] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:58.989300 6570 tablet_bootstrap.cc:492] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: No bootstrap required, opened a new log
I20260810 22:56:58.989506 6570 ts_tablet_manager.cc:1397] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260810 22:56:58.989548 6571 tablet_bootstrap.cc:492] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: No bootstrap required, opened a new log
I20260810 22:56:58.989746 6571 ts_tablet_manager.cc:1397] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:56:58.992393 6570 raft_consensus.cc:348] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:58.992443 6571 raft_consensus.cc:348] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:58.992645 6570 raft_consensus.cc:374] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:58.988099 6573 tablet_bootstrap.cc:492] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Bootstrap starting.
I20260810 22:56:58.992702 6570 raft_consensus.cc:729] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741ea328e0e049e49ac12f595b737760, State: Initialized, Role: FOLLOWER
I20260810 22:56:58.992707 6571 raft_consensus.cc:374] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:58.992774 6571 raft_consensus.cc:729] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f693636c83d4d6ebf9f2d7f5c897532, State: Initialized, Role: FOLLOWER
I20260810 22:56:58.993168 6571 consensus_queue.cc:260] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:58.993180 6570 consensus_queue.cc:260] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:58.994200 6570 ts_tablet_manager.cc:1428] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20260810 22:56:58.994204 6571 ts_tablet_manager.cc:1428] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 22:56:58.994575 6385 heartbeater.cc:503] Master 127.29.78.190:36743 was elected leader, sending a full tablet report...
I20260810 22:56:58.994874 6536 heartbeater.cc:503] Master 127.29.78.190:36743 was elected leader, sending a full tablet report...
I20260810 22:56:58.995405 6573 tablet_bootstrap.cc:654] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:58.996505 6573 log.cc:824] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Log is configured to *not* fsync() on all Append() calls
I20260810 22:56:58.998549 6573 tablet_bootstrap.cc:492] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: No bootstrap required, opened a new log
I20260810 22:56:58.998803 6573 ts_tablet_manager.cc:1397] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260810 22:56:59.001577 6573 raft_consensus.cc:348] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.001804 6573 raft_consensus.cc:374] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.001883 6573 raft_consensus.cc:729] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96656987869c403a84c841a55b052f41, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.002307 6573 consensus_queue.cc:260] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.002872 6230 heartbeater.cc:503] Master 127.29.78.190:36743 was elected leader, sending a full tablet report...
I20260810 22:56:59.003219 6573 ts_tablet_manager.cc:1428] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260810 22:56:59.014389 6575 raft_consensus.cc:482] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:59.014555 6575 raft_consensus.cc:504] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.015547 6575 leader_election.cc:288] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721), 96656987869c403a84c841a55b052f41 (127.29.78.129:46563)
I20260810 22:56:59.019583 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.015968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:34357 (local address 127.29.78.130:37721)
0810 22:56:59.016072 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.016075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.016103 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:56:59.016159 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.016161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.016206 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.016288 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:59.016631 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.017280 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.018442 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.019020 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.019159 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.019223 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.019415 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.019492 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:56:59.019584 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.015803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37721 (local address 127.29.78.131:34357)
0810 22:56:59.015915 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.015930 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.016016 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.016307 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.016310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.016326 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:59.016512 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.016520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.017398 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.017402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.018312 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.018321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.019155 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.019157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.019270 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.019308 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.019375 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.019461 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":17}
I20260810 22:56:59.020241 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2970b332b25c434cb650d83d19bc7fd5" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741ea328e0e049e49ac12f595b737760" is_pre_election: true
I20260810 22:56:59.020547 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.015803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46563 (local address 127.29.78.131:40263)
0810 22:56:59.016281 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.016293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.016446 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.016712 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.016716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.016733 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:59.016849 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.016855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.017637 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.017639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.019310 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.019319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.020195 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.020198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.020265 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.020281 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.020336 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.020416 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":143,"spinlock_wait_cycles":15104,"thread_start_us":124,"threads_started":1}
I20260810 22:56:59.020572 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.015851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:40263 (local address 127.29.78.129:46563)
0810 22:56:59.015975 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.015979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.016396 (+ 417us) server_negotiation.cc:407] Connection header received
0810 22:56:59.016553 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.016559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.016615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.016687 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:59.016959 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.017525 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.019450 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.020017 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.020160 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.020202 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.020371 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.020449 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 22:56:59.020545 6331 raft_consensus.cc:2454] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 0.
I20260810 22:56:59.021032 6414 leader_election.cc:302] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f693636c83d4d6ebf9f2d7f5c897532, 741ea328e0e049e49ac12f595b737760; no voters:
I20260810 22:56:59.021095 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2970b332b25c434cb650d83d19bc7fd5" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41" is_pre_election: true
I20260810 22:56:59.021345 6575 raft_consensus.cc:2781] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:59.021417 6575 raft_consensus.cc:482] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:59.021368 6179 raft_consensus.cc:2454] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 0.
I20260810 22:56:59.021445 6575 raft_consensus.cc:3037] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.022679 6575 raft_consensus.cc:504] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.023056 6575 leader_election.cc:288] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 election: Requested vote from peers 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721), 96656987869c403a84c841a55b052f41 (127.29.78.129:46563)
I20260810 22:56:59.023370 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2970b332b25c434cb650d83d19bc7fd5" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741ea328e0e049e49ac12f595b737760"
I20260810 22:56:59.023419 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2970b332b25c434cb650d83d19bc7fd5" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41"
I20260810 22:56:59.023518 6331 raft_consensus.cc:3037] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.023548 6179 raft_consensus.cc:3037] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.024555 6331 raft_consensus.cc:2454] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 1.
I20260810 22:56:59.024670 6179 raft_consensus.cc:2454] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 1.
I20260810 22:56:59.024914 6414 leader_election.cc:302] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f693636c83d4d6ebf9f2d7f5c897532, 741ea328e0e049e49ac12f595b737760; no voters:
I20260810 22:56:59.025305 6575 raft_consensus.cc:2781] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:59.025527 6575 raft_consensus.cc:686] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 LEADER]: Becoming Leader. State: Replica: 4f693636c83d4d6ebf9f2d7f5c897532, State: Running, Role: LEADER
I20260810 22:56:59.025861 6575 consensus_queue.cc:237] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.028903 6008 catalog_manager.cc:5495] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } health_report { overall_health: HEALTHY } } }
I20260810 22:56:59.048534 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.044994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35528 (local address 127.29.78.131:42511)
0810 22:56:59.045126 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.045132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.045144 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:56:59.045314 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.045317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.045386 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.045483 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.045697 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.046231 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.047119 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.047343 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.047456 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.047739 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.048143 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.048266 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.048325 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.048388 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:56:59.053738 6179 raft_consensus.cc:1261] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Refusing update from remote peer 4f693636c83d4d6ebf9f2d7f5c897532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.053740 6331 raft_consensus.cc:1261] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Refusing update from remote peer 4f693636c83d4d6ebf9f2d7f5c897532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.054469 6582 consensus_queue.cc:1035] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.054634 6575 consensus_queue.cc:1035] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.072849 6007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:59.073031 6007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:59.075609 6007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Sending DeleteTablet for 3 replicas of tablet 2970b332b25c434cb650d83d19bc7fd5
I20260810 22:56:59.076426 6311 tablet_service.cc:1514] Processing DeleteTablet for tablet 2970b332b25c434cb650d83d19bc7fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:37978
I20260810 22:56:59.077021 6596 tablet_replica.cc:330] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: stopping tablet replica
I20260810 22:56:59.077037 6158 tablet_service.cc:1514] Processing DeleteTablet for tablet 2970b332b25c434cb650d83d19bc7fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:48396
I20260810 22:56:59.077315 6007 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:56:59.077410 6596 raft_consensus.cc:2229] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260810 22:56:59.077725 6007 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:59.077744 6596 raft_consensus.cc:2258] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.077741 6468 tablet_service.cc:1514] Processing DeleteTablet for tablet 2970b332b25c434cb650d83d19bc7fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35516
I20260810 22:56:59.078204 6597 tablet_replica.cc:330] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: stopping tablet replica
I20260810 22:56:59.078475 6597 raft_consensus.cc:2229] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:59.078727 6597 raft_consensus.cc:2258] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.079034 6598 tablet_replica.cc:330] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: stopping tablet replica
I20260810 22:56:59.079267 6598 raft_consensus.cc:2229] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:59.079324 6596 ts_tablet_manager.cc:1905] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.079638 6598 raft_consensus.cc:2258] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.081086 6597 ts_tablet_manager.cc:1905] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.083212 6158 tablet_service.cc:1467] Processing CreateTablet for tablet e53b44ccc2624971828dca9228d6e1ff (DEFAULT_TABLE table=test-table [id=08bb8665fcc1484a8fe91dfdef2605d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.083498 6158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53b44ccc2624971828dca9228d6e1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.083536 6311 tablet_service.cc:1467] Processing CreateTablet for tablet e53b44ccc2624971828dca9228d6e1ff (DEFAULT_TABLE table=test-table [id=08bb8665fcc1484a8fe91dfdef2605d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.083761 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53b44ccc2624971828dca9228d6e1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.083878 6597 ts_tablet_manager.cc:1918] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:59.083961 6597 log.cc:1196] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/2970b332b25c434cb650d83d19bc7fd5
I20260810 22:56:59.084261 6597 ts_tablet_manager.cc:1939] T 2970b332b25c434cb650d83d19bc7fd5 P 96656987869c403a84c841a55b052f41: Deleting consensus metadata
I20260810 22:56:59.085427 5994 catalog_manager.cc:4841] TS 96656987869c403a84c841a55b052f41 (127.29.78.129:46563): tablet 2970b332b25c434cb650d83d19bc7fd5 (table test-table [id=a82c7a5efde2437a9e6eb7f8f2354d00]) successfully deleted
I20260810 22:56:59.085897 6573 tablet_bootstrap.cc:492] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Bootstrap starting.
I20260810 22:56:59.086000 6570 tablet_bootstrap.cc:492] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Bootstrap starting.
I20260810 22:56:59.086479 6598 ts_tablet_manager.cc:1905] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.086956 6570 tablet_bootstrap.cc:654] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.086956 6573 tablet_bootstrap.cc:654] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.087749 6596 ts_tablet_manager.cc:1918] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:59.087853 6596 log.cc:1196] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/2970b332b25c434cb650d83d19bc7fd5
I20260810 22:56:59.088177 6596 ts_tablet_manager.cc:1939] T 2970b332b25c434cb650d83d19bc7fd5 P 741ea328e0e049e49ac12f595b737760: Deleting consensus metadata
I20260810 22:56:59.088238 6468 tablet_service.cc:1467] Processing CreateTablet for tablet e53b44ccc2624971828dca9228d6e1ff (DEFAULT_TABLE table=test-table [id=08bb8665fcc1484a8fe91dfdef2605d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.088500 6468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e53b44ccc2624971828dca9228d6e1ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.088997 6598 ts_tablet_manager.cc:1918] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:59.089076 6598 log.cc:1196] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/2970b332b25c434cb650d83d19bc7fd5
I20260810 22:56:59.089367 6598 ts_tablet_manager.cc:1939] T 2970b332b25c434cb650d83d19bc7fd5 P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting consensus metadata
I20260810 22:56:59.089370 5995 catalog_manager.cc:4841] TS 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721): tablet 2970b332b25c434cb650d83d19bc7fd5 (table test-table [id=a82c7a5efde2437a9e6eb7f8f2354d00]) successfully deleted
I20260810 22:56:59.090324 6573 tablet_bootstrap.cc:492] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: No bootstrap required, opened a new log
I20260810 22:56:59.090408 6573 ts_tablet_manager.cc:1397] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260810 22:56:59.090389 5993 catalog_manager.cc:4841] TS 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511): tablet 2970b332b25c434cb650d83d19bc7fd5 (table test-table [id=a82c7a5efde2437a9e6eb7f8f2354d00]) successfully deleted
I20260810 22:56:59.090898 6571 tablet_bootstrap.cc:492] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Bootstrap starting.
I20260810 22:56:59.090910 6573 raft_consensus.cc:348] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.091030 6573 raft_consensus.cc:374] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.091066 6573 raft_consensus.cc:729] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96656987869c403a84c841a55b052f41, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.091184 6573 consensus_queue.cc:260] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.091446 6573 ts_tablet_manager.cc:1428] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.091821 6571 tablet_bootstrap.cc:654] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.092897 6571 tablet_bootstrap.cc:492] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: No bootstrap required, opened a new log
I20260810 22:56:59.092971 6571 ts_tablet_manager.cc:1397] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:59.093051 6570 tablet_bootstrap.cc:492] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: No bootstrap required, opened a new log
I20260810 22:56:59.093134 6570 ts_tablet_manager.cc:1397] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 22:56:59.093344 6571 raft_consensus.cc:348] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.093436 6571 raft_consensus.cc:374] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.093465 6571 raft_consensus.cc:729] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f693636c83d4d6ebf9f2d7f5c897532, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.093585 6571 consensus_queue.cc:260] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.093590 6570 raft_consensus.cc:348] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.093696 6570 raft_consensus.cc:374] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.093732 6570 raft_consensus.cc:729] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741ea328e0e049e49ac12f595b737760, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.093842 6571 ts_tablet_manager.cc:1428] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.093878 6570 consensus_queue.cc:260] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.094136 6570 ts_tablet_manager.cc:1428] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260810 22:56:59.158394 6537 tablet.cc:2369] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:59.171801 6575 raft_consensus.cc:482] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:59.171941 6575 raft_consensus.cc:504] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.172400 6575 leader_election.cc:288] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96656987869c403a84c841a55b052f41 (127.29.78.129:46563), 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721)
I20260810 22:56:59.172888 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53b44ccc2624971828dca9228d6e1ff" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41" is_pre_election: true
I20260810 22:56:59.172920 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53b44ccc2624971828dca9228d6e1ff" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741ea328e0e049e49ac12f595b737760" is_pre_election: true
I20260810 22:56:59.173086 6331 raft_consensus.cc:2454] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 0.
I20260810 22:56:59.173096 6179 raft_consensus.cc:2454] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 0.
I20260810 22:56:59.173417 6414 leader_election.cc:302] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f693636c83d4d6ebf9f2d7f5c897532, 741ea328e0e049e49ac12f595b737760; no voters:
I20260810 22:56:59.173663 6575 raft_consensus.cc:2781] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:59.173727 6575 raft_consensus.cc:482] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:59.173768 6575 raft_consensus.cc:3037] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.175136 6575 raft_consensus.cc:504] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.175539 6575 leader_election.cc:288] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 election: Requested vote from peers 96656987869c403a84c841a55b052f41 (127.29.78.129:46563), 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721)
I20260810 22:56:59.175928 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53b44ccc2624971828dca9228d6e1ff" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741ea328e0e049e49ac12f595b737760"
I20260810 22:56:59.175928 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e53b44ccc2624971828dca9228d6e1ff" candidate_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41"
I20260810 22:56:59.176055 6331 raft_consensus.cc:3037] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.176077 6179 raft_consensus.cc:3037] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.177256 6331 raft_consensus.cc:2454] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 1.
I20260810 22:56:59.177721 6414 leader_election.cc:302] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f693636c83d4d6ebf9f2d7f5c897532, 741ea328e0e049e49ac12f595b737760; no voters:
I20260810 22:56:59.177955 6575 raft_consensus.cc:2781] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:59.178056 6575 raft_consensus.cc:686] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 LEADER]: Becoming Leader. State: Replica: 4f693636c83d4d6ebf9f2d7f5c897532, State: Running, Role: LEADER
I20260810 22:56:59.178153 6179 raft_consensus.cc:2454] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f693636c83d4d6ebf9f2d7f5c897532 in term 1.
I20260810 22:56:59.178190 6575 consensus_queue.cc:237] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.180042 6006 catalog_manager.cc:5495] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } health_report { overall_health: HEALTHY } } }
W20260810 22:56:59.199870 6386 tablet.cc:2369] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 22:56:59 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402618, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.190@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:59.220083 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.211307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36743 (local address 127.0.0.1:45342)
0810 22:56:59.211570 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.211585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.211764 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.212046 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.212048 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.212400 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:56:59.212493 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.212498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.213574 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.213579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.214165 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.214174 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.214694 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.214696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.214754 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.215296 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:56:59.215307 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:56:59.216657 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:56:59.218551 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:59.218559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:59.218563 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:59.218779 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:59.219077 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:56:59.219079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:56:59.219080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:56:59.219240 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:56:59.219664 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:56:59.219669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:56:59.219766 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.219854 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.219960 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
I20260810 22:56:59.220225 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.211391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45342 (local address 127.29.78.190:36743)
0810 22:56:59.211535 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.211539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.211702 (+ 163us) server_negotiation.cc:407] Connection header received
0810 22:56:59.211868 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.211872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.211925 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.212018 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:59.212597 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.213453 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.214266 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.214591 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.214698 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.216788 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:59.216792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:59.216794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:59.216818 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:59.218446 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:59.218881 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:59.218884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:59.218886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:59.218973 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:59.219330 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:59.219333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:59.219335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:59.219545 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:59.219656 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.219930 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.220035 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:56:59.224030 6006 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:45342:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 22:56:59.224795 6006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260810 22:56:59.228209 6231 tablet.cc:2369] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:56:59.229617 6158 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce4a97bbe6c4f8baba92dc37632993f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffef764f4ff94f35a0262f476f9e1be2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:59.229631 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce4a97bbe6c4f8baba92dc37632993f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffef764f4ff94f35a0262f476f9e1be2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:59.229986 6158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce4a97bbe6c4f8baba92dc37632993f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.229895 6468 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce4a97bbe6c4f8baba92dc37632993f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffef764f4ff94f35a0262f476f9e1be2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 22:56:59.230049 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce4a97bbe6c4f8baba92dc37632993f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.230238 6468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce4a97bbe6c4f8baba92dc37632993f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.232453 6570 tablet_bootstrap.cc:492] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: Bootstrap starting.
I20260810 22:56:59.232815 6573 tablet_bootstrap.cc:492] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: Bootstrap starting.
I20260810 22:56:59.232837 6571 tablet_bootstrap.cc:492] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: Bootstrap starting.
I20260810 22:56:59.233429 6570 tablet_bootstrap.cc:654] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.233646 6571 tablet_bootstrap.cc:654] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.233701 6573 tablet_bootstrap.cc:654] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.234709 6570 tablet_bootstrap.cc:492] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: No bootstrap required, opened a new log
I20260810 22:56:59.234741 6571 tablet_bootstrap.cc:492] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: No bootstrap required, opened a new log
I20260810 22:56:59.234799 6570 ts_tablet_manager.cc:1397] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:59.234817 6571 ts_tablet_manager.cc:1397] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:59.235209 6571 raft_consensus.cc:348] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.235272 6570 raft_consensus.cc:348] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.235347 6571 raft_consensus.cc:374] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.235376 6570 raft_consensus.cc:374] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.235394 6571 raft_consensus.cc:729] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f693636c83d4d6ebf9f2d7f5c897532, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.235410 6570 raft_consensus.cc:729] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741ea328e0e049e49ac12f595b737760, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.235518 6571 consensus_queue.cc:260] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.235540 6570 consensus_queue.cc:260] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.235759 6571 ts_tablet_manager.cc:1428] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.235812 6570 ts_tablet_manager.cc:1428] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:56:59.236011 6578 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.236146 6578 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:59.236127 6575 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.236248 6575 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:59.236804 6573 tablet_bootstrap.cc:492] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: No bootstrap required, opened a new log
I20260810 22:56:59.236892 6573 ts_tablet_manager.cc:1397] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 22:56:59.237342 6573 raft_consensus.cc:348] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.237460 6573 raft_consensus.cc:374] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.237496 6573 raft_consensus.cc:729] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96656987869c403a84c841a55b052f41, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.237641 6573 consensus_queue.cc:260] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.237908 6573 ts_tablet_manager.cc:1428] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.237914 6580 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.238353 6580 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:59.367704 6578 raft_consensus.cc:482] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:59.367951 6578 raft_consensus.cc:504] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.369155 6578 leader_election.cc:288] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96656987869c403a84c841a55b052f41 (127.29.78.129:46563), 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511)
I20260810 22:56:59.372845 6567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.369585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:50417 (local address 127.29.78.129:46563)
0810 22:56:59.369701 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.369705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.369867 (+ 162us) server_negotiation.cc:407] Connection header received
0810 22:56:59.370037 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.370040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.370091 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.370167 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:59.370400 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.370916 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.371967 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.372326 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.372442 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.372493 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.372656 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.372721 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 22:56:59.372865 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.369392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46563 (local address 127.29.78.130:50417)
0810 22:56:59.369763 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.369774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.369927 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.370179 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.370181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.370195 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:59.370308 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.370313 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.371012 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.371017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.371852 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.371861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.372443 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.372446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.372541 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.372563 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.372638 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.372702 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":40,"thread_start_us":115,"threads_started":1}
I20260810 22:56:59.373006 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.369511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:47191 (local address 127.29.78.131:42511)
0810 22:56:59.369643 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.369647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.369993 (+ 346us) server_negotiation.cc:407] Connection header received
0810 22:56:59.370120 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.370123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.370171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.370241 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:59.370486 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.370911 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.372038 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.372417 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.372535 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.372584 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.372779 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.372847 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:56:59.373034 6615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.369392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42511 (local address 127.29.78.130:47191)
0810 22:56:59.369899 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.369911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.370024 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.370262 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.370265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.370280 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:59.370387 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.370392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.371010 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.371014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.371929 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.371938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.372528 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.372531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.372619 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.372637 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.372763 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.372829 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":89,"thread_start_us":70,"threads_started":1}
I20260810 22:56:59.373351 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce4a97bbe6c4f8baba92dc37632993f" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41" is_pre_election: true
I20260810 22:56:59.373522 6179 raft_consensus.cc:2454] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 0.
I20260810 22:56:59.373507 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce4a97bbe6c4f8baba92dc37632993f" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" is_pre_election: true
I20260810 22:56:59.373765 6489 raft_consensus.cc:2454] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 0.
I20260810 22:56:59.373896 6259 leader_election.cc:302] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741ea328e0e049e49ac12f595b737760, 96656987869c403a84c841a55b052f41; no voters:
I20260810 22:56:59.374176 6578 raft_consensus.cc:2781] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:59.374257 6578 raft_consensus.cc:482] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:59.374297 6578 raft_consensus.cc:3037] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.375393 6578 raft_consensus.cc:504] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.375723 6578 leader_election.cc:288] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 election: Requested vote from peers 96656987869c403a84c841a55b052f41 (127.29.78.129:46563), 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511)
I20260810 22:56:59.376045 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce4a97bbe6c4f8baba92dc37632993f" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41"
I20260810 22:56:59.376185 6179 raft_consensus.cc:3037] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.377132 6179 raft_consensus.cc:2454] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 1.
I20260810 22:56:59.377408 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce4a97bbe6c4f8baba92dc37632993f" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f693636c83d4d6ebf9f2d7f5c897532"
I20260810 22:56:59.377442 6259 leader_election.cc:302] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741ea328e0e049e49ac12f595b737760, 96656987869c403a84c841a55b052f41; no voters:
I20260810 22:56:59.377539 6489 raft_consensus.cc:3037] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.377660 6578 raft_consensus.cc:2781] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:59.377871 6578 raft_consensus.cc:686] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [term 1 LEADER]: Becoming Leader. State: Replica: 741ea328e0e049e49ac12f595b737760, State: Running, Role: LEADER
I20260810 22:56:59.378115 6578 consensus_queue.cc:237] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.378746 6489 raft_consensus.cc:2454] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 1.
I20260810 22:56:59.379375 6616 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: TxnStatusTablet state changed. Reason: New leader 741ea328e0e049e49ac12f595b737760. Latest consensus state: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.379480 6616 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:59.379912 6618 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 22:56:59.380066 6618 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 22:56:59.380501 6007 catalog_manager.cc:5495] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 reported cstate change: term changed from 0 to 1, leader changed from <none> to 741ea328e0e049e49ac12f595b737760 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:59.426373 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.422760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.29.78.130:37721)
0810 22:56:59.422945 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.422949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.422963 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:59.423029 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.423031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.423083 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.423215 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.423549 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.424055 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.424918 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.425131 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.425265 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.425581 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.425989 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.426109 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.426185 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.426234 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 22:56:59.426379 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.422356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37721 (local address 127.0.0.1:37994)
0810 22:56:59.422554 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.422568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.422658 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.423235 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.423239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.423247 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:59.423405 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.423412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.424183 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.424189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.424786 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.424795 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.425265 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.425269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.425385 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.425428 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.426088 (+ 660us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.426105 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.426175 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.426260 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 22:56:59.429450 6179 raft_consensus.cc:1261] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Refusing update from remote peer 741ea328e0e049e49ac12f595b737760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.429644 6489 raft_consensus.cc:1261] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Refusing update from remote peer 741ea328e0e049e49ac12f595b737760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.429931 6616 consensus_queue.cc:1035] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.430286 6578 consensus_queue.cc:1035] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.431300 6580 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: TxnStatusTablet state changed. Reason: New leader 741ea328e0e049e49ac12f595b737760. Latest consensus state: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.431411 6580 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:59.432116 6575 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: TxnStatusTablet state changed. Reason: New leader 741ea328e0e049e49ac12f595b737760. Latest consensus state: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.432247 6575 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:59.434156 6578 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.434293 6578 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:59.434512 6587 mvcc.cc:205] Tried to move back new op lower bound from 7317105129177882624 to 7317105128975175680. Current Snapshot: MvccSnapshot[applied={T|T < 7317105129177882624}]
I20260810 22:56:59.435298 6616 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.435406 6616 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 741ea328e0e049e49ac12f595b737760: This TxnStatusTablet replica's current role is: LEADER
I20260810 22:56:59.436093 6586 mvcc.cc:205] Tried to move back new op lower bound from 7317105129177882624 to 7317105128975175680. Current Snapshot: MvccSnapshot[applied={T|T < 7317105129177882624}]
I20260810 22:56:59.436206 6580 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.436295 6580 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 96656987869c403a84c841a55b052f41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:59.436605 6575 tablet_replica.cc:439] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } } }
I20260810 22:56:59.436736 6575 tablet_replica.cc:442] T 3ce4a97bbe6c4f8baba92dc37632993f P 4f693636c83d4d6ebf9f2d7f5c897532: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 22:56:59.436844 6585 mvcc.cc:205] Tried to move back new op lower bound from 7317105129177882624 to 7317105128975175680. Current Snapshot: MvccSnapshot[applied={T|T < 7317105129177882624 or (T in {7317105129177882624})}]
I20260810 22:56:59.449671 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.446492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:33207 (local address 127.29.78.130:37721)
0810 22:56:59.446618 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.446622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.446795 (+ 173us) server_negotiation.cc:407] Connection header received
0810 22:56:59.446915 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.446917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.446955 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.447024 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.447272 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.447746 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.448788 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.448985 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.449063 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.449342 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.449398 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.449442 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.449546 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.449582 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:56:59.449756 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.446397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37721 (local address 127.29.78.131:33207)
0810 22:56:59.446708 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.446723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.446829 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.447050 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.447053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.447061 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:59.447172 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.447179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.447837 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.447840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.448684 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.448694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.449092 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.449096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.449177 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.449223 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.449469 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.449475 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.449541 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.449620 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":111,"threads_started":1}
I20260810 22:56:59.456013 6331 raft_consensus.cc:1261] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Refusing update from remote peer 4f693636c83d4d6ebf9f2d7f5c897532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.456312 6179 raft_consensus.cc:1261] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Refusing update from remote peer 4f693636c83d4d6ebf9f2d7f5c897532: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.456329 6582 consensus_queue.cc:1035] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.456670 6575 consensus_queue.cc:1035] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.476835 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.472627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:39535 (local address 127.29.78.131:42511)
0810 22:56:59.472737 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.472742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.472908 (+ 166us) server_negotiation.cc:407] Connection header received
0810 22:56:59.473044 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.473047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.473108 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.473180 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.473591 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.474310 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.475498 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.475804 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.475901 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.476200 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.476277 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.476354 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.476650 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.476710 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:56:59.477329 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.472537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42511 (local address 127.29.78.130:39535)
0810 22:56:59.472800 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.472813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.472951 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.473189 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.473192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.473199 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:59.473476 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.473484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.474438 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.474441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.475347 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.475358 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.475915 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.475919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.476001 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.476051 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.476530 (+ 479us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.476538 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.477060 (+ 522us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.477198 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260810 22:56:59.503070 6008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:59.503268 6008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:59.505268 6008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Sending DeleteTablet for 3 replicas of tablet e53b44ccc2624971828dca9228d6e1ff
I20260810 22:56:59.505808 6311 tablet_service.cc:1514] Processing DeleteTablet for tablet e53b44ccc2624971828dca9228d6e1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:37978
I20260810 22:56:59.505889 6468 tablet_service.cc:1514] Processing DeleteTablet for tablet e53b44ccc2624971828dca9228d6e1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35516
I20260810 22:56:59.506036 6596 tablet_replica.cc:330] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: stopping tablet replica
I20260810 22:56:59.506017 6158 tablet_service.cc:1514] Processing DeleteTablet for tablet e53b44ccc2624971828dca9228d6e1ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:48396
I20260810 22:56:59.506054 6598 tablet_replica.cc:330] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: stopping tablet replica
I20260810 22:56:59.506139 6596 raft_consensus.cc:2229] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:59.506160 6597 tablet_replica.cc:330] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: stopping tablet replica
I20260810 22:56:59.506160 6598 raft_consensus.cc:2229] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:59.506238 6596 raft_consensus.cc:2258] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.506238 6597 raft_consensus.cc:2229] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:59.506330 6597 raft_consensus.cc:2258] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.506323 6598 raft_consensus.cc:2258] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.507119 6598 ts_tablet_manager.cc:1905] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.507344 6596 ts_tablet_manager.cc:1905] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.510403 6598 ts_tablet_manager.cc:1918] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:59.510526 6598 log.cc:1196] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/e53b44ccc2624971828dca9228d6e1ff
I20260810 22:56:59.510628 6597 ts_tablet_manager.cc:1905] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.510720 6598 ts_tablet_manager.cc:1939] T e53b44ccc2624971828dca9228d6e1ff P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting consensus metadata
I20260810 22:56:59.511435 5993 catalog_manager.cc:4841] TS 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511): tablet e53b44ccc2624971828dca9228d6e1ff (table test-table [id=08bb8665fcc1484a8fe91dfdef2605d0]) successfully deleted
I20260810 22:56:59.513406 6597 ts_tablet_manager.cc:1918] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:59.513477 6597 log.cc:1196] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/e53b44ccc2624971828dca9228d6e1ff
I20260810 22:56:59.513715 6597 ts_tablet_manager.cc:1939] T e53b44ccc2624971828dca9228d6e1ff P 96656987869c403a84c841a55b052f41: Deleting consensus metadata
I20260810 22:56:59.514153 6596 ts_tablet_manager.cc:1918] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:56:59.514258 6596 log.cc:1196] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/e53b44ccc2624971828dca9228d6e1ff
I20260810 22:56:59.514529 6596 ts_tablet_manager.cc:1939] T e53b44ccc2624971828dca9228d6e1ff P 741ea328e0e049e49ac12f595b737760: Deleting consensus metadata
I20260810 22:56:59.514575 5994 catalog_manager.cc:4841] TS 96656987869c403a84c841a55b052f41 (127.29.78.129:46563): tablet e53b44ccc2624971828dca9228d6e1ff (table test-table [id=08bb8665fcc1484a8fe91dfdef2605d0]) successfully deleted
I20260810 22:56:59.515187 5995 catalog_manager.cc:4841] TS 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721): tablet e53b44ccc2624971828dca9228d6e1ff (table test-table [id=08bb8665fcc1484a8fe91dfdef2605d0]) successfully deleted
Aug 10 22:56:59 dist-test-slave-6nmv krb5kdc[5960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 22:56:59 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:56:59 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:56:59.537727 6059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:56:59.531313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.29.78.190:36743)
0810 22:56:59.531448 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.531452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.531703 (+ 251us) server_negotiation.cc:407] Connection header received
0810 22:56:59.531809 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.531812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.531867 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.531953 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:59.532632 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.533401 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.534217 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.534543 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.534636 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.537619 (+ 2983us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45348: BlockingRecv error: recv got EOF from 127.0.0.1:45348 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 22:56:59 dist-test-slave-6nmv krb5kdc[5960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM
I20260810 22:56:59.546093 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.539076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.29.78.190:36743)
0810 22:56:59.539171 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.539174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.539276 (+ 102us) server_negotiation.cc:407] Connection header received
0810 22:56:59.539349 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.539352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.539395 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.539472 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:56:59.540028 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.540620 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.541290 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.541463 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.541525 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.543486 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:56:59.543488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:56:59.543490 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:56:59.543508 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:56:59.544647 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:59.545033 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:59.545035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:59.545036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:59.545102 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:56:59.545406 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:56:59.545408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:56:59.545409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:56:59.545555 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:56:59.545647 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.545875 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.545970 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:56:59.548364 6008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:56:59.548796 6008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:59.553272 6158 tablet_service.cc:1467] Processing CreateTablet for tablet 85afed010548400ea1455fe5c933d058 (DEFAULT_TABLE table=test-table [id=1d90837b8dc74cd4b8b19ff9adaf7de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.553407 6311 tablet_service.cc:1467] Processing CreateTablet for tablet 85afed010548400ea1455fe5c933d058 (DEFAULT_TABLE table=test-table [id=1d90837b8dc74cd4b8b19ff9adaf7de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.553579 6158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85afed010548400ea1455fe5c933d058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.553653 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85afed010548400ea1455fe5c933d058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.555912 6573 tablet_bootstrap.cc:492] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: Bootstrap starting.
I20260810 22:56:59.555912 6570 tablet_bootstrap.cc:492] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: Bootstrap starting.
I20260810 22:56:59.556067 6468 tablet_service.cc:1467] Processing CreateTablet for tablet 85afed010548400ea1455fe5c933d058 (DEFAULT_TABLE table=test-table [id=1d90837b8dc74cd4b8b19ff9adaf7de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.556358 6468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85afed010548400ea1455fe5c933d058. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.556793 6570 tablet_bootstrap.cc:654] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.556793 6573 tablet_bootstrap.cc:654] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.558208 6573 tablet_bootstrap.cc:492] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: No bootstrap required, opened a new log
I20260810 22:56:59.558213 6570 tablet_bootstrap.cc:492] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: No bootstrap required, opened a new log
I20260810 22:56:59.558292 6570 ts_tablet_manager.cc:1397] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:59.558301 6573 ts_tablet_manager.cc:1397] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:59.558502 6571 tablet_bootstrap.cc:492] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: Bootstrap starting.
I20260810 22:56:59.558686 6570 raft_consensus.cc:348] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.558791 6570 raft_consensus.cc:374] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.558821 6570 raft_consensus.cc:729] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741ea328e0e049e49ac12f595b737760, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.558964 6570 consensus_queue.cc:260] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.559128 6573 raft_consensus.cc:348] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.559212 6570 ts_tablet_manager.cc:1428] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:56:59.559244 6573 raft_consensus.cc:374] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.559276 6573 raft_consensus.cc:729] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96656987869c403a84c841a55b052f41, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.559399 6571 tablet_bootstrap.cc:654] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.559404 6573 consensus_queue.cc:260] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.559676 6573 ts_tablet_manager.cc:1428] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 22:56:59.560611 6571 tablet_bootstrap.cc:492] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: No bootstrap required, opened a new log
I20260810 22:56:59.560709 6571 ts_tablet_manager.cc:1397] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:59.561125 6571 raft_consensus.cc:348] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.561234 6571 raft_consensus.cc:374] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.561271 6571 raft_consensus.cc:729] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f693636c83d4d6ebf9f2d7f5c897532, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.561403 6571 consensus_queue.cc:260] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.561638 6571 ts_tablet_manager.cc:1428] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.677630 6628 raft_consensus.cc:482] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:59.677767 6628 raft_consensus.cc:504] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.678226 6628 leader_election.cc:288] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96656987869c403a84c841a55b052f41 (127.29.78.129:46563), 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511)
I20260810 22:56:59.678635 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85afed010548400ea1455fe5c933d058" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41" is_pre_election: true
I20260810 22:56:59.678655 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85afed010548400ea1455fe5c933d058" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" is_pre_election: true
I20260810 22:56:59.678783 6179 raft_consensus.cc:2454] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 0.
I20260810 22:56:59.678828 6489 raft_consensus.cc:2454] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 0.
I20260810 22:56:59.679100 6259 leader_election.cc:302] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741ea328e0e049e49ac12f595b737760, 96656987869c403a84c841a55b052f41; no voters:
I20260810 22:56:59.679338 6628 raft_consensus.cc:2781] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:59.679397 6628 raft_consensus.cc:482] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:59.679453 6628 raft_consensus.cc:3037] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.680522 6628 raft_consensus.cc:504] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.680893 6628 leader_election.cc:288] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 election: Requested vote from peers 96656987869c403a84c841a55b052f41 (127.29.78.129:46563), 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511)
I20260810 22:56:59.681147 6179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85afed010548400ea1455fe5c933d058" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96656987869c403a84c841a55b052f41"
I20260810 22:56:59.681286 6179 raft_consensus.cc:3037] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.681234 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85afed010548400ea1455fe5c933d058" candidate_uuid: "741ea328e0e049e49ac12f595b737760" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f693636c83d4d6ebf9f2d7f5c897532"
I20260810 22:56:59.681385 6489 raft_consensus.cc:3037] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.682379 6179 raft_consensus.cc:2454] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 1.
I20260810 22:56:59.682448 6489 raft_consensus.cc:2454] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741ea328e0e049e49ac12f595b737760 in term 1.
I20260810 22:56:59.682703 6259 leader_election.cc:302] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741ea328e0e049e49ac12f595b737760, 96656987869c403a84c841a55b052f41; no voters:
I20260810 22:56:59.682899 6628 raft_consensus.cc:2781] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:59.682986 6628 raft_consensus.cc:686] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 1 LEADER]: Becoming Leader. State: Replica: 741ea328e0e049e49ac12f595b737760, State: Running, Role: LEADER
I20260810 22:56:59.683113 6628 consensus_queue.cc:237] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.684646 6006 catalog_manager.cc:5495] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 reported cstate change: term changed from 0 to 1, leader changed from <none> to 741ea328e0e049e49ac12f595b737760 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "741ea328e0e049e49ac12f595b737760" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:59.711658 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.708393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.29.78.130:37721)
0810 22:56:59.708509 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.708512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.708524 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:56:59.708700 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.708702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.708742 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.708807 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.709032 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.709473 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.710658 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.710853 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.710945 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.711215 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.711273 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.711317 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.711433 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.711473 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:56:59.713785 6489 raft_consensus.cc:1261] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Refusing update from remote peer 741ea328e0e049e49ac12f595b737760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.713778 6179 raft_consensus.cc:1261] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Refusing update from remote peer 741ea328e0e049e49ac12f595b737760: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.714246 6578 consensus_queue.cc:1035] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.714416 6628 consensus_queue.cc:1035] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.724097 6008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:59.724228 6008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:59.726217 6008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Sending DeleteTablet for 3 replicas of tablet 85afed010548400ea1455fe5c933d058
I20260810 22:56:59.726790 6158 tablet_service.cc:1514] Processing DeleteTablet for tablet 85afed010548400ea1455fe5c933d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:48396
I20260810 22:56:59.726841 6468 tablet_service.cc:1514] Processing DeleteTablet for tablet 85afed010548400ea1455fe5c933d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35516
I20260810 22:56:59.726843 6311 tablet_service.cc:1514] Processing DeleteTablet for tablet 85afed010548400ea1455fe5c933d058 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:37978
I20260810 22:56:59.727001 6597 tablet_replica.cc:330] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: stopping tablet replica
I20260810 22:56:59.727012 6596 tablet_replica.cc:330] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: stopping tablet replica
I20260810 22:56:59.727119 6596 raft_consensus.cc:2229] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:56:59.727115 6597 raft_consensus.cc:2229] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:59.727268 6597 raft_consensus.cc:2258] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.727388 6596 raft_consensus.cc:2258] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.727458 6598 tablet_replica.cc:330] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: stopping tablet replica
I20260810 22:56:59.727604 6598 raft_consensus.cc:2229] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:56:59.727737 6598 raft_consensus.cc:2258] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:56:59.728096 6596 ts_tablet_manager.cc:1905] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.728318 6598 ts_tablet_manager.cc:1905] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.728395 6597 ts_tablet_manager.cc:1905] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:56:59.730505 6598 ts_tablet_manager.cc:1918] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:59.730515 6596 ts_tablet_manager.cc:1918] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:59.730569 6598 log.cc:1196] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/85afed010548400ea1455fe5c933d058
I20260810 22:56:59.730584 6596 log.cc:1196] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/85afed010548400ea1455fe5c933d058
I20260810 22:56:59.730566 6597 ts_tablet_manager.cc:1918] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:56:59.730628 6597 log.cc:1196] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/85afed010548400ea1455fe5c933d058
I20260810 22:56:59.730811 6598 ts_tablet_manager.cc:1939] T 85afed010548400ea1455fe5c933d058 P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting consensus metadata
I20260810 22:56:59.730834 6596 ts_tablet_manager.cc:1939] T 85afed010548400ea1455fe5c933d058 P 741ea328e0e049e49ac12f595b737760: Deleting consensus metadata
I20260810 22:56:59.730858 6597 ts_tablet_manager.cc:1939] T 85afed010548400ea1455fe5c933d058 P 96656987869c403a84c841a55b052f41: Deleting consensus metadata
I20260810 22:56:59.731452 5993 catalog_manager.cc:4841] TS 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511): tablet 85afed010548400ea1455fe5c933d058 (table test-table [id=1d90837b8dc74cd4b8b19ff9adaf7de2]) successfully deleted
I20260810 22:56:59.731506 5995 catalog_manager.cc:4841] TS 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721): tablet 85afed010548400ea1455fe5c933d058 (table test-table [id=1d90837b8dc74cd4b8b19ff9adaf7de2]) successfully deleted
I20260810 22:56:59.732354 6006 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 22:56:59.732514 5994 catalog_manager.cc:4841] TS 96656987869c403a84c841a55b052f41 (127.29.78.129:46563): tablet 85afed010548400ea1455fe5c933d058 (table test-table [id=1d90837b8dc74cd4b8b19ff9adaf7de2]) successfully deleted
W20260810 22:56:59.732796 6006 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:56:59.737248 6311 tablet_service.cc:1467] Processing CreateTablet for tablet ae24909e0e3a4a8db0550487d24e4e9d (DEFAULT_TABLE table=test-table [id=fc3da5e245b44b14be0d61f669dec00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.737262 6158 tablet_service.cc:1467] Processing CreateTablet for tablet ae24909e0e3a4a8db0550487d24e4e9d (DEFAULT_TABLE table=test-table [id=fc3da5e245b44b14be0d61f669dec00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.737495 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24909e0e3a4a8db0550487d24e4e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.737522 6158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24909e0e3a4a8db0550487d24e4e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.739287 6573 tablet_bootstrap.cc:492] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41: Bootstrap starting.
I20260810 22:56:59.739545 6570 tablet_bootstrap.cc:492] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: Bootstrap starting.
I20260810 22:56:59.740135 6573 tablet_bootstrap.cc:654] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.740204 6570 tablet_bootstrap.cc:654] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.741170 6468 tablet_service.cc:1467] Processing CreateTablet for tablet ae24909e0e3a4a8db0550487d24e4e9d (DEFAULT_TABLE table=test-table [id=fc3da5e245b44b14be0d61f669dec00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:56:59.741235 6573 tablet_bootstrap.cc:492] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41: No bootstrap required, opened a new log
I20260810 22:56:59.741354 6573 ts_tablet_manager.cc:1397] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:56:59.741382 6468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae24909e0e3a4a8db0550487d24e4e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:56:59.741686 6570 tablet_bootstrap.cc:492] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: No bootstrap required, opened a new log
I20260810 22:56:59.741752 6570 ts_tablet_manager.cc:1397] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:59.741752 6573 raft_consensus.cc:348] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.741868 6573 raft_consensus.cc:374] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.741899 6573 raft_consensus.cc:729] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96656987869c403a84c841a55b052f41, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.741997 6573 consensus_queue.cc:260] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.742110 6570 raft_consensus.cc:348] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.742197 6570 raft_consensus.cc:374] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.742225 6570 raft_consensus.cc:729] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741ea328e0e049e49ac12f595b737760, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.742247 6573 ts_tablet_manager.cc:1428] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.742328 6570 consensus_queue.cc:260] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.742602 6570 ts_tablet_manager.cc:1428] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.743352 6571 tablet_bootstrap.cc:492] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: Bootstrap starting.
I20260810 22:56:59.744009 6571 tablet_bootstrap.cc:654] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: Neither blocks nor log segments found. Creating new log.
I20260810 22:56:59.745121 6571 tablet_bootstrap.cc:492] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: No bootstrap required, opened a new log
I20260810 22:56:59.745195 6571 ts_tablet_manager.cc:1397] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 22:56:59.745512 6571 raft_consensus.cc:348] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.745609 6571 raft_consensus.cc:374] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:56:59.745635 6571 raft_consensus.cc:729] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f693636c83d4d6ebf9f2d7f5c897532, State: Initialized, Role: FOLLOWER
I20260810 22:56:59.745771 6571 consensus_queue.cc:260] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.746027 6571 ts_tablet_manager.cc:1428] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:56:59.897750 6580 raft_consensus.cc:482] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:56:59.897907 6580 raft_consensus.cc:504] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.899060 6580 leader_election.cc:288] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721), 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511)
I20260810 22:56:59.902817 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.899387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:56027 (local address 127.29.78.131:42511)
0810 22:56:59.899504 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.899507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.899688 (+ 181us) server_negotiation.cc:407] Connection header received
0810 22:56:59.899828 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.899830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.899871 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.899943 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:59.900247 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.900667 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.901866 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.902219 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.902331 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.902382 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.902599 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.902693 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:56:59.902822 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.899346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:56915 (local address 127.29.78.130:37721)
0810 22:56:59.899486 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.899492 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.899625 (+ 133us) server_negotiation.cc:407] Connection header received
0810 22:56:59.899780 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.899782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.899825 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.899915 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:56:59.900192 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.900639 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.901872 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.902262 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.902398 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.902448 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.902629 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.902706 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 22:56:59.902859 6662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.899222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37721 (local address 127.29.78.129:56915)
0810 22:56:59.899538 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.899549 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.899681 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.899927 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.899930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.899944 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:59.900096 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.900102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.900772 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.900775 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.901729 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.901739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.902375 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.902379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.902526 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.902542 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.902615 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.902680 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":161,"spinlock_wait_cycles":30464,"thread_start_us":85,"threads_started":1}
I20260810 22:56:59.902859 6663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.899297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42511 (local address 127.29.78.129:56027)
0810 22:56:59.899602 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.899611 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.899720 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.899962 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.899964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.900000 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:56:59.900144 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.900148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.900764 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.900769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.901717 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.901726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.902331 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.902334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.902423 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.902507 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.902584 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.902661 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":104,"spinlock_wait_cycles":2688,"thread_start_us":56,"threads_started":1}
I20260810 22:56:59.903257 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24909e0e3a4a8db0550487d24e4e9d" candidate_uuid: "96656987869c403a84c841a55b052f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741ea328e0e049e49ac12f595b737760" is_pre_election: true
I20260810 22:56:59.903411 6331 raft_consensus.cc:2454] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96656987869c403a84c841a55b052f41 in term 0.
I20260810 22:56:59.903395 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24909e0e3a4a8db0550487d24e4e9d" candidate_uuid: "96656987869c403a84c841a55b052f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" is_pre_election: true
I20260810 22:56:59.903537 6489 raft_consensus.cc:2454] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96656987869c403a84c841a55b052f41 in term 0.
I20260810 22:56:59.903807 6107 leader_election.cc:302] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741ea328e0e049e49ac12f595b737760, 96656987869c403a84c841a55b052f41; no voters:
I20260810 22:56:59.904124 6580 raft_consensus.cc:2781] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:56:59.904197 6580 raft_consensus.cc:482] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:56:59.904248 6580 raft_consensus.cc:3037] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.905263 6580 raft_consensus.cc:504] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.905642 6580 leader_election.cc:288] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [CANDIDATE]: Term 1 election: Requested vote from peers 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721), 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511)
I20260810 22:56:59.906004 6489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24909e0e3a4a8db0550487d24e4e9d" candidate_uuid: "96656987869c403a84c841a55b052f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f693636c83d4d6ebf9f2d7f5c897532"
I20260810 22:56:59.905983 6331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae24909e0e3a4a8db0550487d24e4e9d" candidate_uuid: "96656987869c403a84c841a55b052f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741ea328e0e049e49ac12f595b737760"
I20260810 22:56:59.906112 6331 raft_consensus.cc:3037] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.906124 6489 raft_consensus.cc:3037] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:56:59.907023 6489 raft_consensus.cc:2454] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96656987869c403a84c841a55b052f41 in term 1.
I20260810 22:56:59.907208 6331 raft_consensus.cc:2454] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96656987869c403a84c841a55b052f41 in term 1.
I20260810 22:56:59.907301 6104 leader_election.cc:302] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f693636c83d4d6ebf9f2d7f5c897532, 96656987869c403a84c841a55b052f41; no voters:
I20260810 22:56:59.907593 6580 raft_consensus.cc:2781] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:56:59.907778 6580 raft_consensus.cc:686] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 1 LEADER]: Becoming Leader. State: Replica: 96656987869c403a84c841a55b052f41, State: Running, Role: LEADER
I20260810 22:56:59.908004 6580 consensus_queue.cc:237] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } }
I20260810 22:56:59.910025 6008 catalog_manager.cc:5495] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96656987869c403a84c841a55b052f41 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "96656987869c403a84c841a55b052f41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96656987869c403a84c841a55b052f41" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20260810 22:56:59.942189 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.938738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.29.78.129:46563)
0810 22:56:59.939064 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.939077 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.939088 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:56:59.939164 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.939165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.939213 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.939352 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.939564 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.940070 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.940815 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.941041 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.941112 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.941350 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.941780 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.941933 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.941958 (+ 25us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.942007 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":125,"threads_started":1}
I20260810 22:56:59.952293 6565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.948692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:35709 (local address 127.29.78.130:37721)
0810 22:56:59.948831 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.948863 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.948988 (+ 125us) server_negotiation.cc:407] Connection header received
0810 22:56:59.949161 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.949166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.949220 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.949333 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.949640 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.950226 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.951135 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.951339 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.951451 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.951810 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.951885 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.951968 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.952088 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.952138 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:56:59.952358 6669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.948565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:37721 (local address 127.29.78.129:35709)
0810 22:56:59.948887 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.948902 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.949057 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.949354 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.949357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.949366 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:59.949522 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.949528 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.950353 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.950357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.951012 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.951021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.951453 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.951456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.951560 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.951611 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.951988 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.951997 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.952071 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.952155 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":103,"threads_started":1}
I20260810 22:56:59.958359 6489 raft_consensus.cc:1261] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Refusing update from remote peer 96656987869c403a84c841a55b052f41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.958443 6331 raft_consensus.cc:1261] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Refusing update from remote peer 96656987869c403a84c841a55b052f41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:56:59.958850 6664 consensus_queue.cc:1035] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f693636c83d4d6ebf9f2d7f5c897532" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.959066 6580 consensus_queue.cc:1035] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741ea328e0e049e49ac12f595b737760" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:56:59.975668 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.972405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:38103 (local address 127.29.78.129:46563)
0810 22:56:59.972556 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 22:56:59.972560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:56:59.972574 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:56:59.972670 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:56:59.972672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:56:59.972722 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:56:59.972811 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:56:59.973061 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.973562 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.974500 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.974794 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.974872 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.975225 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.975302 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.975374 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 22:56:59.975471 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 22:56:59.975515 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 22:56:59.975698 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:56:59.972381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46563 (local address 127.29.78.130:38103)
0810 22:56:59.972497 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 22:56:59.972512 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:56:59.972595 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:56:59.972834 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:56:59.972838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:56:59.972848 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 22:56:59.972951 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.972960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.973657 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.973659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.974363 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:56:59.974373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:56:59.974917 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:56:59.974921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:56:59.975014 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:56:59.975075 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 22:56:59.975395 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 22:56:59.975401 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:56:59.975461 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 22:56:59.975550 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 22:56:59.998282 6006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:56:59.998507 6006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:57:00.000494 6006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ea14fb9c37c4696b768b5af79bfb253: Sending DeleteTablet for 3 replicas of tablet ae24909e0e3a4a8db0550487d24e4e9d
I20260810 22:57:00.001067 6311 tablet_service.cc:1514] Processing DeleteTablet for tablet ae24909e0e3a4a8db0550487d24e4e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:37978
I20260810 22:57:00.001071 6158 tablet_service.cc:1514] Processing DeleteTablet for tablet ae24909e0e3a4a8db0550487d24e4e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:48396
I20260810 22:57:00.001246 6596 tablet_replica.cc:330] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: stopping tablet replica
I20260810 22:57:00.001253 6597 tablet_replica.cc:330] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41: stopping tablet replica
I20260810 22:57:00.001358 6596 raft_consensus.cc:2229] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:57:00.001374 6597 raft_consensus.cc:2229] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:57:00.001466 6596 raft_consensus.cc:2258] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:57:00.001652 6597 raft_consensus.cc:2258] T ae24909e0e3a4a8db0550487d24e4e9d P 96656987869c403a84c841a55b052f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:57:00.001645 6468 tablet_service.cc:1514] Processing DeleteTablet for tablet ae24909e0e3a4a8db0550487d24e4e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:56:59 UTC) from {username='oryx', principal='oryx/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:35516
I20260810 22:57:00.001960 6598 tablet_replica.cc:330] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: stopping tablet replica
I20260810 22:57:00.001953 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 6086
I20260810 22:57:00.002110 6598 raft_consensus.cc:2229] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:57:00.002223 6598 raft_consensus.cc:2258] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:57:00.006124 6596 ts_tablet_manager.cc:1905] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:57:00.007030 6598 ts_tablet_manager.cc:1905] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:57:00.008330 6596 ts_tablet_manager.cc:1918] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:57:00.008409 6596 log.cc:1196] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/ae24909e0e3a4a8db0550487d24e4e9d
I20260810 22:57:00.008652 6596 ts_tablet_manager.cc:1939] T ae24909e0e3a4a8db0550487d24e4e9d P 741ea328e0e049e49ac12f595b737760: Deleting consensus metadata
I20260810 22:57:00.009486 5995 catalog_manager.cc:4841] TS 741ea328e0e049e49ac12f595b737760 (127.29.78.130:37721): tablet ae24909e0e3a4a8db0550487d24e4e9d (table test-table [id=fc3da5e245b44b14be0d61f669dec00c]) successfully deleted
I20260810 22:57:00.009851 6598 ts_tablet_manager.cc:1918] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 22:57:00.009953 6598 log.cc:1196] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/ae24909e0e3a4a8db0550487d24e4e9d
I20260810 22:57:00.010223 6598 ts_tablet_manager.cc:1939] T ae24909e0e3a4a8db0550487d24e4e9d P 4f693636c83d4d6ebf9f2d7f5c897532: Deleting consensus metadata
I20260810 22:57:00.011056 5993 catalog_manager.cc:4841] TS 4f693636c83d4d6ebf9f2d7f5c897532 (127.29.78.131:42511): tablet ae24909e0e3a4a8db0550487d24e4e9d (table test-table [id=fc3da5e245b44b14be0d61f669dec00c]) successfully deleted
W20260810 22:57:00.015290 5994 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:46563 (error 108)
I20260810 22:57:00.016170 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 6240
I20260810 22:57:00.016557 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:00.016117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46563 (local address 127.0.0.1:48416)
0810 22:57:00.016340 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:00.016429 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46563: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
W20260810 22:57:00.017519 5994 catalog_manager.cc:4584] TS 96656987869c403a84c841a55b052f41 (127.29.78.129:46563): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae24909e0e3a4a8db0550487d24e4e9d: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46563: connect: Connection refused (error 111)
I20260810 22:57:00.031167 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 6394
I20260810 22:57:00.041085 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 5975
2026-08-10T22:57:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3186 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:57:00 dist-test-slave-6nmv krb5kdc[6679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:57:00 dist-test-slave-6nmv krb5kdc[6679](info): set up 2 sockets
Aug 10 22:57:00 dist-test-slave-6nmv krb5kdc[6679](info): commencing operation
krb5kdc: starting...
W20260810 22:57:01.095798 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:57:01 dist-test-slave-6nmv krb5kdc[6679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402621, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:57:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:57:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:57:01.212598 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39007
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39007
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:01.317484 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:01.317797 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:01.317896 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:01.321687 6693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:57:01.321769 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:01.321806 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:01.321861 6693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:57:01.321899 6693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:57:01.327060 6693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:01.328094 6693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:01.329041 6693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:01.337384 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:01.337409 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:01.337427 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:01.337530 6693 server_base.cc:1034] running on GCE node
I20260810 22:57:01.338316 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:01.339359 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:01.340547 6693 hybrid_clock.cc:648] HybridClock initialized: now 1786402621340540 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T22:57:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1304 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:57:01 dist-test-slave-6nmv krb5kdc[6708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:57:01 dist-test-slave-6nmv krb5kdc[6708](info): set up 2 sockets
Aug 10 22:57:01 dist-test-slave-6nmv krb5kdc[6708](info): commencing operation
krb5kdc: starting...
W20260810 22:57:02.384625 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:57:02 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:57:02.505103 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46359
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:46359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:02.615142 6725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:02.615391 6725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:02.615445 6725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:02.619253 6725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:57:02.619305 6725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:02.619323 6725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:02.619361 6725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:57:02.619392 6725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:57:02.624488 6725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:46359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:02.625519 6725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:02.626564 6725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:02.634789 6732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:02.634790 6735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:02.634797 6733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:02.634850 6725 server_base.cc:1034] running on GCE node
I20260810 22:57:02.635614 6725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:02.636615 6725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:02.637848 6725 hybrid_clock.cc:648] HybridClock initialized: now 1786402622637800 us; error 26 us; skew 500 ppm
Aug 10 22:57:02 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402622, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:02.646983 6725 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:57:02.647989 6725 webserver.cc:460] Webserver started at http://127.29.78.190:37769/ using document root <none> and password file <none>
I20260810 22:57:02.648561 6725 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:02.648650 6725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:02.648890 6725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:02.650940 6725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "1b410cd9bd9a4224852edda8145b8d74"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "fe26f26eee872a149e18131e2a6962af"
server_key_iv: "80b02829d1ab14392aba2e3fd9385eee"
server_key_version: "encryptionkey@0"
I20260810 22:57:02.651527 6725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "1b410cd9bd9a4224852edda8145b8d74"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "fe26f26eee872a149e18131e2a6962af"
server_key_iv: "80b02829d1ab14392aba2e3fd9385eee"
server_key_version: "encryptionkey@0"
I20260810 22:57:02.655216 6725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:57:02.657495 6743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:02.658740 6725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:57:02.658879 6725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "1b410cd9bd9a4224852edda8145b8d74"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "fe26f26eee872a149e18131e2a6962af"
server_key_iv: "80b02829d1ab14392aba2e3fd9385eee"
server_key_version: "encryptionkey@0"
I20260810 22:57:02.658989 6725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:02.672915 6725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:02.676280 6725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:02.676486 6725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:02.683640 6725 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:46359
I20260810 22:57:02.683656 6802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:46359 every 8 connection(s)
I20260810 22:57:02.684549 6725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:57:02.687352 6803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:57:02.691612 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 6725
I20260810 22:57:02.691784 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:57:02.692066 30010 external_mini_cluster.cc:1428] Setting key d40cd844c4ad003eb432393400434885
I20260810 22:57:02.693192 6803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Bootstrap starting.
I20260810 22:57:02.695732 6803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Neither blocks nor log segments found. Creating new log.
I20260810 22:57:02.696681 6803 log.cc:824] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:02.698781 6803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: No bootstrap required, opened a new log
Aug 10 22:57:02 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:57:02.703171 6803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:02.703509 6803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:57:02.703599 6803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b410cd9bd9a4224852edda8145b8d74, State: Initialized, Role: FOLLOWER
I20260810 22:57:02.704032 6803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:02.704180 6803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:57:02.704264 6803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:57:02.704365 6803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:57:02.705238 6803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:02.705570 6803 leader_election.cc:302] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b410cd9bd9a4224852edda8145b8d74; no voters:
I20260810 22:57:02.705902 6803 leader_election.cc:288] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:57:02.705997 6809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:57:02.706212 6809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 LEADER]: Becoming Leader. State: Replica: 1b410cd9bd9a4224852edda8145b8d74, State: Running, Role: LEADER
I20260810 22:57:02.706698 6809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:02.707245 6803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:57:02.707715 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:02.693460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.29.78.190:46359)
0810 22:57:02.694367 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:02.694394 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:02.694430 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:57:02.695056 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:02.695072 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:02.695394 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:02.695662 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:02.696619 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:02.697870 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:02.698948 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:02.699371 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:02.699481 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:02.702354 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:02.702359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:02.702371 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:02.702397 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:02.704415 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:02.704879 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:02.704885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:02.704890 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:02.705010 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:02.705284 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:02.705288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:02.705290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:02.705755 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:02.705996 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:02.706591 (+ 595us) server_negotiation.cc:299] Negotiation successful
0810 22:57:02.706897 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":631,"threads_started":1}
I20260810 22:57:02.708371 6811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b410cd9bd9a4224852edda8145b8d74. Latest consensus state: current_term: 1 leader_uuid: "1b410cd9bd9a4224852edda8145b8d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } } }
I20260810 22:57:02.708492 6811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:02.708791 6810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b410cd9bd9a4224852edda8145b8d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } } }
I20260810 22:57:02.708904 6810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: This master's current role is: LEADER
W20260810 22:57:02.712949 6824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:57:02.713040 6824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:57:02.713131 6818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:57:02.714051 6818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:57:02.721673 6818 catalog_manager.cc:1347] Generated new cluster ID: 77474e585d9b4ef3a7c43d8331d760f0
I20260810 22:57:02.721760 6818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:57:02.744156 6818 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:57:02.745440 6818 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:57:02.753582 6818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Generated new TSK 0
I20260810 22:57:02.754357 6818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:57:02.792915 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46359
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:02.930763 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:02.931020 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:02.931121 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:02.935182 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:02.935240 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:02.935339 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:57:02.940589 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:02.941722 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:02.942803 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:02.953151 6836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:02.953169 6839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:02.953243 6837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:02.953425 6830 server_base.cc:1034] running on GCE node
I20260810 22:57:02.953933 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:02.954546 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:02.955727 6830 hybrid_clock.cc:648] HybridClock initialized: now 1786402622955719 us; error 28 us; skew 500 ppm
Aug 10 22:57:02 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402622, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:02.958552 6830 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:57:02.959349 6830 webserver.cc:460] Webserver started at http://127.29.78.129:46209/ using document root <none> and password file <none>
I20260810 22:57:02.959968 6830 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:02.960038 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:02.960261 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:02.962447 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "d4941761eb2f4a3e8e3558f5b82bfa36"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "7c043e979c498efd9d4700bd842aec5b"
server_key_iv: "9e991af28752957bc6db95e214108300"
server_key_version: "encryptionkey@0"
I20260810 22:57:02.963032 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "d4941761eb2f4a3e8e3558f5b82bfa36"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "7c043e979c498efd9d4700bd842aec5b"
server_key_iv: "9e991af28752957bc6db95e214108300"
server_key_version: "encryptionkey@0"
I20260810 22:57:02.966672 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:57:02.969034 6847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:02.970316 6830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:57:02.970429 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "d4941761eb2f4a3e8e3558f5b82bfa36"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "7c043e979c498efd9d4700bd842aec5b"
server_key_iv: "9e991af28752957bc6db95e214108300"
server_key_version: "encryptionkey@0"
I20260810 22:57:02.970544 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:03.018954 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:03.021962 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:03.022187 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:03.022816 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:03.023715 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:03.023761 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.023837 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:03.023870 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.033095 6830 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:40413
I20260810 22:57:03.033116 6972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:40413 every 8 connection(s)
I20260810 22:57:03.034075 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:57:03.041054 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 6830
I20260810 22:57:03.041179 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:57:03.041543 30010 external_mini_cluster.cc:1428] Setting key 562e14bdb663a4d7b76d2a97ae00c671
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402622, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:57:03.048062 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.035856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52525 (local address 127.29.78.190:46359)
0810 22:57:03.036002 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:03.036007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:03.036885 (+ 878us) server_negotiation.cc:407] Connection header received
0810 22:57:03.038244 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:03.038247 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:03.038291 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:03.038360 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:03.039397 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.039911 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.040965 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.041336 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.041444 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.043988 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:03.043994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:03.044008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:03.044041 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:03.045646 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:03.046307 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:03.046313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:03.046317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:03.046440 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:03.046792 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:03.046798 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:03.046802 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:03.047073 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:03.047179 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:03.047667 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 22:57:03.047813 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:57:03.048784 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.036266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:52525)
0810 22:57:03.036708 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:03.036770 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:03.038053 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:03.038480 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:03.038486 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:03.038809 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:03.039262 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.039275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.040018 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.040021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.040844 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.040852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.041473 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.041476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.041657 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.042331 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:03.042347 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:03.043824 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:03.045764 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:03.045771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:03.045786 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:03.046211 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:03.046532 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:03.046534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:03.046535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:03.046666 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:03.047176 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:03.047181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:03.047406 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 22:57:03.047555 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 22:57:03.047799 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260810 22:57:03.050093 6973 heartbeater.cc:347] Connected to a master server at 127.29.78.190:46359
I20260810 22:57:03.050326 6973 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:03.050803 6973 heartbeater.cc:511] Master 127.29.78.190:46359 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:57:03.052330 6763 ts_manager.cc:194] Registered new tserver with Master: d4941761eb2f4a3e8e3558f5b82bfa36 (127.29.78.129:40413)
I20260810 22:57:03.053530 6763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:52525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:57:03.069213 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46359
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:03.178310 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:03.178574 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:03.178615 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:03.182425 6980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:03.182475 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:03.182565 6980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:57:03.187573 6980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:03.188612 6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:03.189606 6980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:03.199112 6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:03.199112 6987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:03.199107 6989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:03.199476 6980 server_base.cc:1034] running on GCE node
I20260810 22:57:03.199903 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:03.200557 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:03.201754 6980 hybrid_clock.cc:648] HybridClock initialized: now 1786402623201711 us; error 65 us; skew 500 ppm
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402623, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:03.204552 6980 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:57:03.205523 6980 webserver.cc:460] Webserver started at http://127.29.78.130:37341/ using document root <none> and password file <none>
I20260810 22:57:03.206167 6980 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:03.206264 6980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:03.206506 6980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:03.208819 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "2c1c6e248c7a448e91e42991cb7b2cc2"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "89a00af0de4bd76a61debee74de7a26f"
server_key_iv: "639eeba9909ffb3a246923b787b793ca"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.209481 6980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "2c1c6e248c7a448e91e42991cb7b2cc2"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "89a00af0de4bd76a61debee74de7a26f"
server_key_iv: "639eeba9909ffb3a246923b787b793ca"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.213167 6980 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 22:57:03.215688 6997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.216847 6980 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:57:03.217010 6980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "2c1c6e248c7a448e91e42991cb7b2cc2"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "89a00af0de4bd76a61debee74de7a26f"
server_key_iv: "639eeba9909ffb3a246923b787b793ca"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.217121 6980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:03.253285 6980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:03.256636 6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:03.256875 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:03.257510 6980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:03.258450 6980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:03.258523 6980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.258595 6980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:03.258646 6980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.267655 6980 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:42443
I20260810 22:57:03.267679 7128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:42443 every 8 connection(s)
I20260810 22:57:03.268582 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:57:03.276232 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 6980
I20260810 22:57:03.276367 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:57:03.276726 30010 external_mini_cluster.cc:1428] Setting key a38a20daf461fd404bf494cd67cd8845
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402623, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:57:03.283317 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.270536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:46867 (local address 127.29.78.190:46359)
0810 22:57:03.270680 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:03.270685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:03.271555 (+ 870us) server_negotiation.cc:407] Connection header received
0810 22:57:03.273046 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:03.273049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:03.273099 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:03.273181 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:03.274484 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.275119 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.276235 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.276594 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.276707 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.279558 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:03.279564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:03.279570 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:03.279601 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:03.281126 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:03.281760 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:03.281765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:03.281767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:03.281898 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:03.282239 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:03.282243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:03.282245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:03.282433 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:03.282555 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:03.283042 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 22:57:03.283166 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 22:57:03.284130 7131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.270961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:46867)
0810 22:57:03.271400 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:03.271442 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:03.272801 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:03.273336 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:03.273342 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:03.273775 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:03.274352 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.274366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.275255 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.275257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.276096 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.276105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.276771 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.276774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.276953 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.277719 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:03.277739 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:03.279387 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:03.281245 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:03.281252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:03.281267 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:03.281648 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:03.281989 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:03.281994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:03.282003 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:03.282142 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:03.282555 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:03.282563 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:03.282807 (+ 244us) client_negotiation.cc:769] Sending connection context
0810 22:57:03.283008 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 22:57:03.283312 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260810 22:57:03.285380 7129 heartbeater.cc:347] Connected to a master server at 127.29.78.190:46359
I20260810 22:57:03.285622 7129 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:03.286151 7129 heartbeater.cc:511] Master 127.29.78.190:46359 requested a full tablet report, sending...
I20260810 22:57:03.287279 6763 ts_manager.cc:194] Registered new tserver with Master: 2c1c6e248c7a448e91e42991cb7b2cc2 (127.29.78.130:42443)
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:57:03.287982 6763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:46867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:57:03.304901 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46359
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:03.416869 7136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:03.417125 7136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:03.417166 7136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:03.421064 7136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:03.421108 7136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:03.421159 7136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:57:03.425695 7136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:03.426625 7136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:03.427592 7136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:03.437256 7144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:03.437326 7136 server_base.cc:1034] running on GCE node
W20260810 22:57:03.437278 7147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:03.437271 7143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:03.437875 7136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:03.438529 7136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:03.439683 7136 hybrid_clock.cc:648] HybridClock initialized: now 1786402623439656 us; error 42 us; skew 500 ppm
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402623, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:03.442452 7136 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:57:03.443318 7136 webserver.cc:460] Webserver started at http://127.29.78.131:33779/ using document root <none> and password file <none>
I20260810 22:57:03.443840 7136 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:03.443940 7136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:03.444164 7136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:03.446122 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "d15985df23c3427bb800af2a718bf396"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "d0f286458034eaca8a5460888f850562"
server_key_iv: "8047df13a72cfcc02223cd6f4c14ca3b"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.446640 7136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "d15985df23c3427bb800af2a718bf396"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "d0f286458034eaca8a5460888f850562"
server_key_iv: "8047df13a72cfcc02223cd6f4c14ca3b"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.450204 7136 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 22:57:03.452513 7154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.453692 7136 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:57:03.453895 7136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "d15985df23c3427bb800af2a718bf396"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "d0f286458034eaca8a5460888f850562"
server_key_iv: "8047df13a72cfcc02223cd6f4c14ca3b"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.454008 7136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:03.463300 7136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:03.465969 7136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:03.466144 7136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:03.466662 7136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:03.467455 7136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:03.467527 7136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.467597 7136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:03.467651 7136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.477047 7136 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:44339
I20260810 22:57:03.477079 7285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:44339 every 8 connection(s)
I20260810 22:57:03.477995 7136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:57:03.481900 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 7136
I20260810 22:57:03.482052 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:57:03.482429 30010 external_mini_cluster.cc:1428] Setting key fad8ac6faa1ec0e0a07e4aa2a5af2f48
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402623, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:57:03.491127 6806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.479769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:49781 (local address 127.29.78.190:46359)
0810 22:57:03.479905 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:03.479909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:03.480657 (+ 748us) server_negotiation.cc:407] Connection header received
0810 22:57:03.482175 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:03.482179 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:03.482240 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:03.482332 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:03.483693 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.484411 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.485527 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.485736 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.485870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.488186 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:03.488189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:03.488191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:03.488214 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:03.489399 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:03.489939 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:03.489942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:03.489943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:03.490015 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:03.490286 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:03.490288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:03.490289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:03.490446 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:03.490514 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:03.490925 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 22:57:03.491016 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 22:57:03.491954 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.480077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:49781)
0810 22:57:03.480512 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:03.480553 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:03.481804 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:03.482491 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:03.482501 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:03.482848 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:03.483459 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.483475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.484550 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.484554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.485418 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.485427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.485893 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.485896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.486028 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.486564 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:03.486579 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:03.488048 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:03.489516 (+ 1468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:03.489520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:03.489548 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:03.489786 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:03.490085 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:03.490087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:03.490088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:03.490206 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:03.490524 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:03.490528 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:03.490724 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 22:57:03.490856 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 22:57:03.491084 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":103,"threads_started":1}
I20260810 22:57:03.493103 7286 heartbeater.cc:347] Connected to a master server at 127.29.78.190:46359
I20260810 22:57:03.493340 7286 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:03.493778 7286 heartbeater.cc:511] Master 127.29.78.190:46359 requested a full tablet report, sending...
I20260810 22:57:03.494791 6763 ts_manager.cc:194] Registered new tserver with Master: d15985df23c3427bb800af2a718bf396 (127.29.78.131:44339)
I20260810 22:57:03.495512 6763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:49781
I20260810 22:57:03.496158 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
Entry for principal oryx/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab.
WARNING: no policy specified for oryx/127.29.78.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
Entry for principal oryx/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab.
WARNING: no policy specified for oryx/127.29.78.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
Entry for principal oryx/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab.
I20260810 22:57:03.567853 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 6830
I20260810 22:57:03.575991 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 6980
I20260810 22:57:03.584398 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 7136
I20260810 22:57:03.592139 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 6725
I20260810 22:57:03.600217 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46359
--webserver_interface=127.29.78.190
--webserver_port=37769
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:46359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:03.709617 7299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:03.709877 7299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:03.709921 7299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:03.713637 7299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:57:03.713682 7299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:03.713699 7299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:03.713732 7299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:57:03.713744 7299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:57:03.718717 7299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:46359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:46359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=37769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:03.719609 7299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:03.720533 7299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:03.728801 7309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:03.728786 7305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:03.728771 7307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:03.729198 7299 server_base.cc:1034] running on GCE node
I20260810 22:57:03.729790 7299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:03.730875 7299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:03.732066 7299 hybrid_clock.cc:648] HybridClock initialized: now 1786402623732056 us; error 29 us; skew 500 ppm
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402623, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:03.734862 7299 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:57:03.735706 7299 webserver.cc:460] Webserver started at http://127.29.78.190:37769/ using document root <none> and password file <none>
I20260810 22:57:03.736248 7299 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:03.736363 7299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:03.740121 7299 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 22:57:03.742309 7316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.743461 7299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:57:03.743599 7299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "1b410cd9bd9a4224852edda8145b8d74"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "fe26f26eee872a149e18131e2a6962af"
server_key_iv: "80b02829d1ab14392aba2e3fd9385eee"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.744122 7299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:03.755947 7299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:03.759037 7299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:03.759258 7299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:03.766629 7299 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:46359
I20260810 22:57:03.766664 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:46359 every 8 connection(s)
I20260810 22:57:03.767724 7299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:57:03.772020 7374 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:57:03.773820 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Bootstrap starting.
I20260810 22:57:03.777138 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 7299
I20260810 22:57:03.777606 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:40413
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:46359
--webserver_port=46209
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:57:03.795357 7374 log.cc:824] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:03.803683 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:57:03.804108 7374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Bootstrap complete.
I20260810 22:57:03.809602 7374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:03.810169 7374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b410cd9bd9a4224852edda8145b8d74, State: Initialized, Role: FOLLOWER
I20260810 22:57:03.810680 7374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:03.810792 7374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:57:03.810879 7374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:57:03.810979 7374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:57:03.813105 7374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:03.813431 7374 leader_election.cc:302] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b410cd9bd9a4224852edda8145b8d74; no voters:
I20260810 22:57:03.813747 7374 leader_election.cc:288] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:57:03.813892 7378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:57:03.814126 7378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [term 2 LEADER]: Becoming Leader. State: Replica: 1b410cd9bd9a4224852edda8145b8d74, State: Running, Role: LEADER
I20260810 22:57:03.814406 7378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } }
I20260810 22:57:03.814930 7374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:57:03.815681 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1b410cd9bd9a4224852edda8145b8d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } } }
I20260810 22:57:03.815794 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:03.816071 7380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b410cd9bd9a4224852edda8145b8d74. Latest consensus state: current_term: 2 leader_uuid: "1b410cd9bd9a4224852edda8145b8d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b410cd9bd9a4224852edda8145b8d74" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 46359 } } }
I20260810 22:57:03.816143 7380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74 [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:03.816534 7386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:57:03.819756 7386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:57:03.820314 7386 catalog_manager.cc:1259] Loaded cluster ID: 77474e585d9b4ef3a7c43d8331d760f0
I20260810 22:57:03.820453 7386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:57:03.822685 7386 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:57:03.823338 7386 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1b410cd9bd9a4224852edda8145b8d74: Loaded TSK: 0
I20260810 22:57:03.824505 7386 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:57:03.900238 7376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:03.900473 7376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:03.900517 7376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:03.904650 7376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:03.904706 7376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:03.904760 7376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:57:03.909886 7376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.129.keytab
--principal=oryx/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:40413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=46209
--tserver_master_addrs=127.29.78.190:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:03.910887 7376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:03.911919 7376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:03.921530 7402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:03.921653 7403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:03.921553 7406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:03.921820 7376 server_base.cc:1034] running on GCE node
I20260810 22:57:03.922304 7376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:03.922999 7376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:03.924187 7376 hybrid_clock.cc:648] HybridClock initialized: now 1786402623924161 us; error 47 us; skew 500 ppm
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402623, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:03.927177 7376 init.cc:377] Logged in from keytab as oryx/127.29.78.129@KRBTEST.COM (short username oryx)
I20260810 22:57:03.928126 7376 webserver.cc:460] Webserver started at http://127.29.78.129:46209/ using document root <none> and password file <none>
I20260810 22:57:03.928665 7376 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:03.928774 7376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:03.932633 7376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:57:03.934692 7414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.936008 7376 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:57:03.936132 7376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "d4941761eb2f4a3e8e3558f5b82bfa36"
format_stamp: "Formatted at 2026-08-10 22:57:02 on dist-test-slave-6nmv"
server_key: "7c043e979c498efd9d4700bd842aec5b"
server_key_iv: "9e991af28752957bc6db95e214108300"
server_key_version: "encryptionkey@0"
I20260810 22:57:03.936651 7376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:03.952235 7376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:03.955612 7376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:03.955829 7376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:03.956435 7376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:03.957279 7376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:03.957327 7376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.957394 7376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:03.957425 7376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:03.966512 7376 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:40413
I20260810 22:57:03.966543 7538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:40413 every 8 connection(s)
I20260810 22:57:03.967497 7376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:57:03.975571 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 7376
I20260810 22:57:03.976125 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:42443
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:46359
--webserver_port=37341
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:57:03.981117 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:03.969593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33843 (local address 127.29.78.190:46359)
0810 22:57:03.970103 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:03.970122 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:03.970448 (+ 326us) server_negotiation.cc:407] Connection header received
0810 22:57:03.971982 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:03.971993 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:03.972271 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:03.972549 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:03.973589 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.974664 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.975828 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.976532 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.976720 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.980446 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:33843: BlockingRecv error: recv got EOF from 127.29.78.129:33843 (error 108)
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":150,"threads_started":1}
I20260810 22:57:03.981338 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.969668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:33843)
0810 22:57:03.970247 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:03.970289 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:03.971543 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:03.972630 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:03.972636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:03.973008 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:03.973442 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.973456 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.974794 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.974796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.975696 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.975711 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.976703 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.976706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.976875 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.977514 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:03.977531 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:03.980274 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":208,"threads_started":1}
W20260810 22:57:03.981904 7539 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:46359 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.78.190:46359: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:03.987288 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.982326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:59981)
0810 22:57:03.982439 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:03.982455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:03.982568 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:03.982787 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:03.982790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:03.982940 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:03.983033 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.983038 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.983685 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.983689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.984205 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.984212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.984652 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.984656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.984747 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.985218 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:03.985230 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:03.987153 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 22:57:03.987381 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:03.982454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:59981 (local address 127.29.78.190:46359)
0810 22:57:03.982593 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:03.982598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:03.982625 (+ 27us) server_negotiation.cc:407] Connection header received
0810 22:57:03.982663 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:03.982665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:03.982699 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:03.982775 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:03.983138 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.983586 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.984322 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.984531 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.984640 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.987268 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:59981: BlockingRecv error: recv got EOF from 127.29.78.129:59981 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:03 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:03.992404 7542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:03.987808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:33463)
0810 22:57:03.987914 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:03.987929 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:03.988044 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:03.988271 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:03.988273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:03.988412 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:03.988470 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.988475 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.989124 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.989128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.989588 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:03.989594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.990071 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.990075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:03.990159 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.990603 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:03.990614 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:03.992281 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 22:57:03.992487 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:03.987934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33463 (local address 127.29.78.190:46359)
0810 22:57:03.988102 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:03.988107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:03.988118 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:57:03.988152 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:03.988154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:03.988183 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:03.988258 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:03.988577 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.988991 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.989691 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:03.989961 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:03.990060 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:03.992384 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:33463: BlockingRecv error: recv got EOF from 127.29.78.129:33463 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 22:57:03.992648 7539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:57:04.088299 7543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:04.088567 7543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:04.088610 7543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:04.092644 7543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:04.092705 7543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:04.092813 7543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:57:04.098083 7543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.130.keytab
--principal=oryx/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:42443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=37341
--tserver_master_addrs=127.29.78.190:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:04.099134 7543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:04.100157 7543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:04.109489 7549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:04.109637 7543 server_base.cc:1034] running on GCE node
W20260810 22:57:04.109519 7551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:04.109536 7548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:04.110095 7543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:04.110711 7543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:04.111867 7543 hybrid_clock.cc:648] HybridClock initialized: now 1786402624111856 us; error 23 us; skew 500 ppm
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402624, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:04.114797 7543 init.cc:377] Logged in from keytab as oryx/127.29.78.130@KRBTEST.COM (short username oryx)
I20260810 22:57:04.115751 7543 webserver.cc:460] Webserver started at http://127.29.78.130:37341/ using document root <none> and password file <none>
I20260810 22:57:04.116290 7543 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:04.116430 7543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:04.120852 7543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 22:57:04.123057 7559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:04.124327 7543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:57:04.124430 7543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "2c1c6e248c7a448e91e42991cb7b2cc2"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "89a00af0de4bd76a61debee74de7a26f"
server_key_iv: "639eeba9909ffb3a246923b787b793ca"
server_key_version: "encryptionkey@0"
I20260810 22:57:04.124914 7543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:04.142092 7543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:04.144892 7543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:04.145040 7543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:04.145571 7543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:04.146418 7543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:04.146464 7543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:04.146502 7543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:04.146515 7543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:04.155294 7543 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:42443
I20260810 22:57:04.155390 7685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:42443 every 8 connection(s)
I20260810 22:57:04.156289 7543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:57:04.164542 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 7543
I20260810 22:57:04.165083 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:44339
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:46359
--webserver_port=33779
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:57:04.167722 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:04.158155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:35603 (local address 127.29.78.190:46359)
0810 22:57:04.158295 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.158306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.159127 (+ 821us) server_negotiation.cc:407] Connection header received
0810 22:57:04.160439 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.160442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.160520 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.160623 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.161772 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.162321 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.163477 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.163716 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.163848 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.167597 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:35603: BlockingRecv error: recv got EOF from 127.29.78.130:35603 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:57:04.168357 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.158535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:35603)
0810 22:57:04.158974 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:04.159023 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:04.160220 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:04.160797 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:04.160803 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:04.161179 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:04.161625 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.161638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.162478 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.162487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.163349 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.163358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.163893 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.163897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.164082 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.164679 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:04.164701 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:04.167462 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":115,"threads_started":1}
W20260810 22:57:04.168905 7686 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:46359 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.78.190:46359: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:04.175238 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.169374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:39585)
0810 22:57:04.169484 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:04.169507 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:04.169582 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:04.169915 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:04.169917 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:04.170047 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:04.170136 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.170141 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.171314 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.171320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.171880 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.171888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.172548 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.172551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.172621 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.173160 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:04.173170 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:04.175148 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 22:57:04.175625 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:04.169515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:39585 (local address 127.29.78.190:46359)
0810 22:57:04.169652 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.169658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.169675 (+ 17us) server_negotiation.cc:407] Connection header received
0810 22:57:04.169723 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.169727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.169781 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.169896 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.170256 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.171160 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.172001 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.172427 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.172557 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.175468 (+ 2911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:39585: BlockingRecv error: recv got EOF from 127.29.78.130:39585 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:04.181221 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.176067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:51569)
0810 22:57:04.176247 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:04.176264 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:04.176357 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:04.176624 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:04.176630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:04.176823 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:04.176962 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.176973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.177656 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.177661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.178566 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.178576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.178975 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.178979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.179060 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.179469 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:04.179483 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:04.181110 (+ 1627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107}
W20260810 22:57:04.181388 7686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:57:04.181524 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:04.176138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:51569 (local address 127.29.78.190:46359)
0810 22:57:04.176244 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.176247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.176310 (+ 63us) server_negotiation.cc:407] Connection header received
0810 22:57:04.176469 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.176471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.176512 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.176574 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.177078 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.177547 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.178678 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.178876 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.178950 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.181208 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:51569: BlockingRecv error: recv got EOF from 127.29.78.130:51569 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 22:57:04.276974 7690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:04.277213 7690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:04.277254 7690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:04.281069 7690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:04.281118 7690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:04.281168 7690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:57:04.285916 7690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/krb5kdc/oryx_127.29.78.131.keytab
--principal=oryx/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:44339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=33779
--tserver_master_addrs=127.29.78.190:46359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:04.286845 7690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:04.287839 7690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:04.296933 7698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:04.297000 7690 server_base.cc:1034] running on GCE node
W20260810 22:57:04.296913 7701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:04.296913 7699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:04.297649 7690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:04.298264 7690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:04.299432 7690 hybrid_clock.cc:648] HybridClock initialized: now 1786402624299416 us; error 26 us; skew 500 ppm
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402624, etypes {rep=17 tkt=17 ses=17}, oryx/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:04.302139 7690 init.cc:377] Logged in from keytab as oryx/127.29.78.131@KRBTEST.COM (short username oryx)
I20260810 22:57:04.302960 7690 webserver.cc:460] Webserver started at http://127.29.78.131:33779/ using document root <none> and password file <none>
I20260810 22:57:04.303520 7690 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:04.303622 7690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:04.307435 7690 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 22:57:04.309453 7710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:04.310673 7690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:57:04.310801 7690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "d15985df23c3427bb800af2a718bf396"
format_stamp: "Formatted at 2026-08-10 22:57:03 on dist-test-slave-6nmv"
server_key: "d0f286458034eaca8a5460888f850562"
server_key_iv: "8047df13a72cfcc02223cd6f4c14ca3b"
server_key_version: "encryptionkey@0"
I20260810 22:57:04.311313 7690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:04.323483 7690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:04.326182 7690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:04.326370 7690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:04.326921 7690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:04.327706 7690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:04.327791 7690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:04.327853 7690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:04.327904 7690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:04.337148 7690 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:44339
I20260810 22:57:04.337170 7847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:44339 every 8 connection(s)
I20260810 22:57:04.338272 7690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:57:04.342628 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 7690
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 22:57:04.350594 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:04.339942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:57233 (local address 127.29.78.190:46359)
0810 22:57:04.340100 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.340105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.341141 (+ 1036us) server_negotiation.cc:407] Connection header received
0810 22:57:04.343128 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.343131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.343195 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.343286 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.344504 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.345135 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.346366 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.346685 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.346796 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.350480 (+ 3684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:57233: BlockingRecv error: recv got EOF from 127.29.78.131:57233 (error 108)
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":49}
I20260810 22:57:04.351279 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.340356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:57233)
0810 22:57:04.340954 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:04.341006 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:04.342909 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:04.343481 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:04.343489 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:04.343935 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:04.344365 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.344385 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.345247 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.345251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.346258 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.346268 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.346914 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.346921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.347108 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.347718 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:04.347734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:04.350400 (+ 2666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":134,"threads_started":1}
W20260810 22:57:04.351722 7848 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:46359 (0 consecutive failures): Not authorized: Failed to ping master at 127.29.78.190:46359: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
I20260810 22:57:04.352792 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.343659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.29.78.190:46359)
0810 22:57:04.343900 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.343903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.343912 (+ 9us) server_negotiation.cc:407] Connection header received
0810 22:57:04.343940 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.343941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.343972 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.344070 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.344868 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.345428 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.346270 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.346655 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.346726 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.348354 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:04.348362 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:04.348380 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:04.348412 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:04.350598 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:04.351139 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:04.351145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:04.351150 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:04.351308 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:04.351622 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:04.351625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:04.351626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:04.352013 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:04.352232 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:04.352467 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 22:57:04.352647 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":175,"thread_start_us":91,"threads_started":1}
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:04.357196 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.352128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:32855)
0810 22:57:04.352677 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:04.352692 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:04.352760 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:04.352972 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:04.352976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:04.353135 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:04.353214 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.353219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.354018 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.354021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.354667 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.354676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.355124 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.355127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.355189 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.355571 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:04.355580 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:04.357069 (+ 1489us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476}
W20260810 22:57:04.357278 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:04.352189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:32855 (local address 127.29.78.190:46359)
0810 22:57:04.352292 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.352295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.352761 (+ 466us) server_negotiation.cc:407] Connection header received
0810 22:57:04.352821 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.352827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.352873 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.352953 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.353324 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.353918 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.354771 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.355027 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.355116 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.357174 (+ 2058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:32855: BlockingRecv error: recv got EOF from 127.29.78.131:32855 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:04.362042 7851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.357698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:52413)
0810 22:57:04.357795 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:04.357849 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:04.357927 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:04.358149 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:04.358152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:04.358289 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:04.358349 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.358353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.359009 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.359012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.359461 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.359467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.359877 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.359880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.359953 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.360319 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:04.360329 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:04.361901 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 22:57:04.362118 7541 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:04.357790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:52413 (local address 127.29.78.190:46359)
0810 22:57:04.357943 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.357949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.357965 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:57:04.357999 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.358002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.358035 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.358136 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.358452 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.358903 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.359565 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.359773 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.359873 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.362010 (+ 2137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:52413: BlockingRecv error: recv got EOF from 127.29.78.131:52413 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260810 22:57:04.362267 7848 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:04 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:04.998529 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:04.993282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:50395)
0810 22:57:04.993584 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:04.993598 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:04.993690 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:04.993961 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:04.993964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:04.994144 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:04.994244 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.994250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.994978 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.994980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.995529 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:04.995537 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.995996 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.995998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:04.996063 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.996547 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:04.996558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:04.998407 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
W20260810 22:57:04.998680 7856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:04.993402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:50395 (local address 127.29.78.190:46359)
0810 22:57:04.993646 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:04.993650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:04.993663 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:57:04.993773 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:04.993775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:04.993870 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:04.993943 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:04.994346 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.994885 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.995637 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:04.995894 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:04.995976 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:04.998548 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:50395: BlockingRecv error: recv got EOF from 127.29.78.129:50395 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":64,"threads_started":1}
Aug 10 22:57:05 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:05 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:05.187587 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:05.182242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:49741)
0810 22:57:05.182523 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:05.182536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:05.182626 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:05.182842 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:05.182844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:05.183016 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:05.183129 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:05.183133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.183844 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.183846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:05.184355 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:05.184362 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.184838 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.184842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:05.184957 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:05.185514 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:05.185526 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:05.187404 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":82,"threads_started":1}
W20260810 22:57:05.187646 7856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:05.182308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49741 (local address 127.29.78.190:46359)
0810 22:57:05.182439 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:05.182442 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:05.182594 (+ 152us) server_negotiation.cc:407] Connection header received
0810 22:57:05.182695 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:05.182697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:05.182738 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:05.182805 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:05.183215 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.183761 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.184493 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.184722 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.184832 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:05.187539 (+ 2707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:49741: BlockingRecv error: recv got EOF from 127.29.78.130:49741 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 22:57:05 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:05 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:05.367955 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:05.362862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:38249)
0810 22:57:05.363127 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:05.363139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:05.363227 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:05.363436 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:05.363438 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:05.363606 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:05.363692 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:05.363696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.364310 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.364313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:05.364796 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:05.364802 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.365191 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.365195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:05.365270 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:05.365898 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:05.365912 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:05.367831 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
W20260810 22:57:05.368085 7856 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:05.362946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38249 (local address 127.29.78.190:46359)
0810 22:57:05.363095 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:05.363098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:05.363206 (+ 108us) server_negotiation.cc:407] Connection header received
0810 22:57:05.363298 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:05.363300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:05.363346 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:05.363409 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:05.363779 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.364225 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.364885 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:05.365072 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:05.365156 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:05.367964 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:38249: BlockingRecv error: recv got EOF from 127.29.78.131:38249 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 22:57:06 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:06 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:06.004629 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:05.999557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:43873)
0810 22:57:05.999853 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:05.999868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:05.999960 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:06.000208 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:06.000210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:06.000404 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:06.000498 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:06.000503 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.001132 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.001134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:06.001633 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:06.001640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.002079 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.002082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:06.002146 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:06.002744 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:06.002755 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:06.004500 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":117,"threads_started":1}
W20260810 22:57:06.004740 7860 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:05.999669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:43873 (local address 127.29.78.190:46359)
0810 22:57:05.999914 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:05.999917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:05.999932 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:57:06.000053 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:06.000055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:06.000095 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:06.000173 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:06.000589 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.001047 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.001738 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.001955 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.002053 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:06.004622 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:43873: BlockingRecv error: recv got EOF from 127.29.78.129:43873 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
Aug 10 22:57:06 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:06 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:06.193619 7861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:06.188479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:52919)
0810 22:57:06.188761 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:06.188773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:06.188868 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:06.189083 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:06.189085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:06.189260 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:06.189355 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:06.189360 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.190088 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.190090 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:06.190634 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:06.190641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.191024 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.191026 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:06.191085 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:06.191667 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:06.191679 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:06.193500 (+ 1821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":99,"threads_started":1}
W20260810 22:57:06.193853 7860 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:06.188530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:52919 (local address 127.29.78.190:46359)
0810 22:57:06.188691 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:06.188694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:06.188821 (+ 127us) server_negotiation.cc:407] Connection header received
0810 22:57:06.188942 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:06.188944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:06.188986 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:06.189042 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:06.189454 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.189999 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.190730 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.190933 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.191004 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:06.193696 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:52919: BlockingRecv error: recv got EOF from 127.29.78.130:52919 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 22:57:06 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:06 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:06.374768 7862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:06.368845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:46887)
0810 22:57:06.369204 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:06.369221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:06.369336 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:06.369622 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:06.369625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:06.369894 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:06.370007 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:06.370013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.370775 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.370780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:06.371302 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:06.371310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.371895 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.371899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:06.372012 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:06.372660 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:06.372678 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:06.374625 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
W20260810 22:57:06.374845 7860 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:06.369035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46887 (local address 127.29.78.190:46359)
0810 22:57:06.369187 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:06.369192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:06.369279 (+ 87us) server_negotiation.cc:407] Connection header received
0810 22:57:06.369447 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:06.369450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:06.369507 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:06.369597 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:06.370120 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.370679 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.371422 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:06.371753 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:06.371865 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:06.374754 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:46887: BlockingRecv error: recv got EOF from 127.29.78.131:46887 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 22:57:07 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:07 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:07.010785 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:07.005373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:33371)
0810 22:57:07.005736 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:07.005752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:07.005895 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:07.006242 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:07.006244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:07.006473 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:07.006588 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:07.006594 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.007303 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.007305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:07.007855 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:07.007862 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.008285 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.008287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:07.008359 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:07.008824 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:07.008835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:07.010671 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
W20260810 22:57:07.010933 7864 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:07.005468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33371 (local address 127.29.78.190:46359)
0810 22:57:07.005804 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:07.005811 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:07.005842 (+ 31us) server_negotiation.cc:407] Connection header received
0810 22:57:07.006028 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:07.006030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:07.006094 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:07.006185 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:07.006700 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.007203 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.007968 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.008180 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.008252 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:07.010802 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:33371: BlockingRecv error: recv got EOF from 127.29.78.129:33371 (error 108)
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
Aug 10 22:57:07 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:07 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:07.199674 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:07.194457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:46815)
0810 22:57:07.194727 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:07.194740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:07.194836 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:07.195067 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:07.195069 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:07.195251 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:07.195348 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:07.195353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.196054 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.196056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:07.196605 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:07.196612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.197052 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.197055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:07.197117 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:07.197704 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:07.197716 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:07.199552 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":96,"threads_started":1}
W20260810 22:57:07.199847 7864 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:07.194612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:46815 (local address 127.29.78.190:46359)
0810 22:57:07.194759 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:07.194763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:07.194790 (+ 27us) server_negotiation.cc:407] Connection header received
0810 22:57:07.194924 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:07.194926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:07.194969 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:07.195033 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:07.195456 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.195963 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.196710 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.196931 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.197035 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:07.199691 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:46815: BlockingRecv error: recv got EOF from 127.29.78.130:46815 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 22:57:07 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:07 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:07.381206 7867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:07.375574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:45651)
0810 22:57:07.375865 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:07.375879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:07.375979 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:07.376210 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:07.376212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:07.376388 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:07.376481 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:07.376486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.377190 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.377192 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:07.377739 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:07.377747 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.378333 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.378335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:07.378408 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:07.379145 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:07.379167 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:07.381090 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
W20260810 22:57:07.381363 7864 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:07.375630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:45651 (local address 127.29.78.190:46359)
0810 22:57:07.375805 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:07.375810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:07.375954 (+ 144us) server_negotiation.cc:407] Connection header received
0810 22:57:07.376060 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:07.376062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:07.376108 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:07.376175 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:07.376586 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.377083 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.377869 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:07.378222 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:07.378319 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:07.381226 (+ 2907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:45651: BlockingRecv error: recv got EOF from 127.29.78.131:45651 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 22:57:08 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:08 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:08.020269 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:08.011570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:49065)
0810 22:57:08.011905 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:08.011919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:08.012015 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:08.012271 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:08.012273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:08.012465 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:08.012570 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:08.012575 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.013300 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.013302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:08.013868 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:08.013876 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.014312 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.014314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:08.014380 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:08.018478 (+ 4098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:08.018490 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:08.020157 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":93,"threads_started":1}
W20260810 22:57:08.020468 7869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:08.011722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:49065 (local address 127.29.78.190:46359)
0810 22:57:08.011997 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:08.012000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:08.012014 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:57:08.012106 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:08.012108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:08.012153 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:08.012231 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:08.012679 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.013208 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.013963 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.014218 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.014294 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:08.020294 (+ 6000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:49065: BlockingRecv error: recv got EOF from 127.29.78.129:49065 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
Aug 10 22:57:08 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:08 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:08.206410 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:08.200942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:34671)
0810 22:57:08.201236 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:08.201249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:08.201402 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:08.201727 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:08.201730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:08.201985 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:08.202101 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:08.202109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.202927 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.202933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:08.203461 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:08.203469 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.203955 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.203959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:08.204053 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:08.204537 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:08.204549 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:08.206303 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":107,"threads_started":1}
W20260810 22:57:08.206602 7869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:08.201011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34671 (local address 127.29.78.190:46359)
0810 22:57:08.201180 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:08.201185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:08.201334 (+ 149us) server_negotiation.cc:407] Connection header received
0810 22:57:08.201527 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:08.201532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:08.201588 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:08.201701 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:08.202235 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.202796 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.203596 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.203818 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.203939 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:08.206462 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:34671: BlockingRecv error: recv got EOF from 127.29.78.130:34671 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 22:57:08 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:08 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:08.388088 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:08.382213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:35687)
0810 22:57:08.382507 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:08.382521 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:08.382646 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:08.382973 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:08.382976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:08.383208 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:08.383304 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:08.383309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.384130 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.384133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:08.384973 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:08.384987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.385543 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.385545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:08.385607 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:08.386105 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:08.386116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:08.387975 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
W20260810 22:57:08.388231 7869 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:08.382273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:35687 (local address 127.29.78.190:46359)
0810 22:57:08.382442 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:08.382447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:08.382596 (+ 149us) server_negotiation.cc:407] Connection header received
0810 22:57:08.382795 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:08.382798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:08.382852 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:08.382952 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:08.383425 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.383989 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.385096 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:08.385429 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:08.385526 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:08.388111 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:35687: BlockingRecv error: recv got EOF from 127.29.78.131:35687 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 22:57:09 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:09 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:09.026470 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:09.021235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:40361)
0810 22:57:09.021522 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:09.021538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:09.021667 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:09.022014 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:09.022016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:09.022294 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:09.022414 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:09.022431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.023129 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.023131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:09.023634 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:09.023640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.024042 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.024044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:09.024096 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:09.024577 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:09.024588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:09.026357 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":97,"threads_started":1}
W20260810 22:57:09.026618 7874 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:09.021256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:40361 (local address 127.29.78.190:46359)
0810 22:57:09.021526 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:09.021532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:09.021610 (+ 78us) server_negotiation.cc:407] Connection header received
0810 22:57:09.021760 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:09.021763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:09.021846 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:09.021965 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:09.022523 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.023043 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.023723 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.023937 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.024014 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:09.026490 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:40361: BlockingRecv error: recv got EOF from 127.29.78.129:40361 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":112,"threads_started":1}
Aug 10 22:57:09 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:09 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:09.212677 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:09.207176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:56509)
0810 22:57:09.207543 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:09.207559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:09.207701 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:09.207973 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:09.207976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:09.208201 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:09.208290 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:09.208295 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.209052 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.209057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:09.209620 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:09.209628 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.210117 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.210121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:09.210202 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:09.210655 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:09.210665 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:09.212465 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":111,"threads_started":1}
W20260810 22:57:09.212733 7874 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:09.207328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:56509 (local address 127.29.78.190:46359)
0810 22:57:09.207518 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:09.207523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:09.207644 (+ 121us) server_negotiation.cc:407] Connection header received
0810 22:57:09.207801 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:09.207805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:09.207860 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:09.207949 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:09.208408 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.208934 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.209733 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.210008 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.210110 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:09.212627 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:56509: BlockingRecv error: recv got EOF from 127.29.78.130:56509 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 22:57:09 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:09 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:09.394528 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:09.389026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:53295)
0810 22:57:09.389297 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:09.389310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:09.389405 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:09.389725 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:09.389729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:09.390009 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:09.390131 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:09.390137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.391073 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.391075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:09.391550 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:09.391556 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.391970 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.391973 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:09.392034 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:09.392538 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:09.392549 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:09.394419 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":94,"threads_started":1}
W20260810 22:57:09.394762 7874 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:09.389102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:53295 (local address 127.29.78.190:46359)
0810 22:57:09.389248 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:09.389252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:09.389381 (+ 129us) server_negotiation.cc:407] Connection header received
0810 22:57:09.389549 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:09.389551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:09.389595 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:09.389665 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:09.390296 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.390982 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.391668 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:09.391878 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:09.391965 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:09.394530 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:53295: BlockingRecv error: recv got EOF from 127.29.78.131:53295 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 22:57:10 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:10 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:10.033488 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:10.027503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:34905)
0810 22:57:10.027835 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:10.027853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:10.027977 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:10.028340 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:10.028343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:10.028574 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:10.028673 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:10.028679 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.029617 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.029622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:10.030219 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:10.030228 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.030714 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.030718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:10.030805 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:10.031352 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:10.031364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:10.033298 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":87,"threads_started":1}
W20260810 22:57:10.033550 7878 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:10.027560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:34905 (local address 127.29.78.190:46359)
0810 22:57:10.027798 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:10.027802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:10.027945 (+ 143us) server_negotiation.cc:407] Connection header received
0810 22:57:10.028151 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:10.028154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:10.028220 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:10.028320 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:10.028795 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.029506 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.030343 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.030594 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.030707 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:10.033427 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:34905: BlockingRecv error: recv got EOF from 127.29.78.129:34905 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":62,"threads_started":1}
Aug 10 22:57:10 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:10 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:10.219789 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:10.213478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:54317)
0810 22:57:10.213907 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:10.213932 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:10.214094 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:10.214456 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:10.214459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:10.214701 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:10.214810 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:10.214817 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.215764 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.215769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:10.216463 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:10.216473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.217005 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.217009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:10.217105 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:10.217657 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:10.217669 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:10.219670 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":118,"threads_started":1}
W20260810 22:57:10.219944 7878 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:10.213620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:54317 (local address 127.29.78.190:46359)
0810 22:57:10.213842 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:10.213848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:10.214027 (+ 179us) server_negotiation.cc:407] Connection header received
0810 22:57:10.214212 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:10.214217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:10.214303 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:10.214434 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:10.214962 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.215650 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.216591 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.216862 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.216999 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:10.219809 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:54317: BlockingRecv error: recv got EOF from 127.29.78.130:54317 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 22:57:10 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:10 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:10.401075 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:10.395305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:60301)
0810 22:57:10.395690 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:10.395703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:10.395844 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:10.396130 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:10.396133 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:10.396358 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:10.396445 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:10.396450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.397222 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.397227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:10.397798 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:10.397805 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.398290 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.398294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:10.398379 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:10.398965 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:10.398978 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:10.400968 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":150,"threads_started":1}
W20260810 22:57:10.401257 7878 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:10.395409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60301 (local address 127.29.78.190:46359)
0810 22:57:10.395624 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:10.395630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:10.395782 (+ 152us) server_negotiation.cc:407] Connection header received
0810 22:57:10.395957 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:10.395961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:10.396018 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:10.396107 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:10.396581 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.397112 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.397944 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:10.398169 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:10.398274 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:10.401122 (+ 2848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:60301: BlockingRecv error: recv got EOF from 127.29.78.131:60301 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Aug 10 22:57:11 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:11 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:11.040097 7881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:11.034310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:43579)
0810 22:57:11.034655 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:11.034682 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:11.034785 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:11.035143 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:11.035147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:11.035383 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:11.035510 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:11.035524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.036235 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.036240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:11.036772 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:11.036779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.037222 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.037225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:11.037306 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:11.037911 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:11.037924 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:11.039927 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":70,"threads_started":1}
W20260810 22:57:11.040201 7882 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:11.034487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:43579 (local address 127.29.78.190:46359)
0810 22:57:11.034756 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:11.034760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:11.034773 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:57:11.034932 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:11.034935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:11.034989 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:11.035104 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:11.035631 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.036133 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.036888 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.037110 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.037216 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:11.040065 (+ 2849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:43579: BlockingRecv error: recv got EOF from 127.29.78.129:43579 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
Aug 10 22:57:11 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:11 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:11.226642 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:11.220823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:36035)
0810 22:57:11.221204 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:11.221217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:11.221319 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:11.221574 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:11.221576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:11.221773 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:11.221933 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:11.221939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.222656 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.222658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:11.223187 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:11.223193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.223639 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.223641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:11.223700 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:11.224264 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:11.224275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:11.226462 (+ 2187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":122,"threads_started":1}
W20260810 22:57:11.226719 7882 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:11.220838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:36035 (local address 127.29.78.190:46359)
0810 22:57:11.221027 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:11.221033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:11.221370 (+ 337us) server_negotiation.cc:407] Connection header received
0810 22:57:11.221413 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:11.221415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:11.221462 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:11.221531 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:11.222057 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.222561 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.223323 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.223538 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.223621 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:11.226625 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:36035: BlockingRecv error: recv got EOF from 127.29.78.130:36035 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 22:57:11 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:11 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:11.407949 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:11.401993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:54631)
0810 22:57:11.402423 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:11.402436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:11.402586 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:11.402855 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:11.402857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:11.403087 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:11.403175 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:11.403181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.403941 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.403947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:11.404491 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:11.404498 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.404918 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.404922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:11.405000 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:11.405547 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:11.405558 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:11.407762 (+ 2204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":111,"threads_started":1}
W20260810 22:57:11.408069 7882 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:11.402159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:54631 (local address 127.29.78.190:46359)
0810 22:57:11.402355 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:11.402361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:11.402525 (+ 164us) server_negotiation.cc:407] Connection header received
0810 22:57:11.402686 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:11.402689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:11.402743 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:11.402829 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:11.403303 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.403828 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.404601 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:11.404810 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:11.404911 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:11.407960 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:54631: BlockingRecv error: recv got EOF from 127.29.78.131:54631 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 10 22:57:12 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:12 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:12.046051 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:12.040891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:52569)
0810 22:57:12.041158 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:12.041173 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:12.041262 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:12.041469 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:12.041471 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:12.041641 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:12.041732 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:12.041737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.042471 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.042473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:12.042998 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:12.043005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.043377 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.043379 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:12.043435 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:12.043982 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:12.044002 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:12.045874 (+ 1872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":113,"threads_started":1}
W20260810 22:57:12.046137 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:12.041062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52569 (local address 127.29.78.190:46359)
0810 22:57:12.041277 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:12.041280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:12.041291 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:57:12.041322 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:12.041324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:12.041363 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:12.041434 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:12.041864 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.042376 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.043092 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.043289 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.043358 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:12.045999 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:52569: BlockingRecv error: recv got EOF from 127.29.78.129:52569 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
Aug 10 22:57:12 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:12 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:12.233778 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:12.227830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:49705)
0810 22:57:12.228198 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:12.228214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:12.228372 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:12.228696 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:12.228699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:12.228930 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:12.229023 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:12.229029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.230085 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.230090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:12.230643 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:12.230651 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.231118 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.231121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:12.231207 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:12.231730 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:12.231741 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:12.233604 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":129,"threads_started":1}
W20260810 22:57:12.233852 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:12.227969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49705 (local address 127.29.78.190:46359)
0810 22:57:12.228202 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:12.228209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:12.228309 (+ 100us) server_negotiation.cc:407] Connection header received
0810 22:57:12.228506 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:12.228508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:12.228581 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:12.228678 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:12.229145 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.229968 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.230763 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.231004 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.231112 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:12.233725 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:49705: BlockingRecv error: recv got EOF from 127.29.78.130:49705 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Aug 10 22:57:12 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:12 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:12.415134 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:12.408783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:55307)
0810 22:57:12.409165 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:12.409184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:12.409379 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:12.409686 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:12.409689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:12.409954 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:12.410062 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:12.410068 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.410913 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.410920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:12.411514 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:12.411523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.412067 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.412072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:12.412167 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:12.412773 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:12.412786 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:12.415009 (+ 2223us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":90,"threads_started":1}
W20260810 22:57:12.415278 7886 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:12.408967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:55307 (local address 127.29.78.190:46359)
0810 22:57:12.409149 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:12.409155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:12.409282 (+ 127us) server_negotiation.cc:407] Connection header received
0810 22:57:12.409492 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:12.409497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:12.409562 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:12.409662 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:12.410202 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.410774 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.411672 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:12.411933 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:12.412060 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:12.415151 (+ 3091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:55307: BlockingRecv error: recv got EOF from 127.29.78.131:55307 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 22:57:13 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:13 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:13.058959 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:13.047036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:46495)
0810 22:57:13.047383 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:13.047403 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:13.047555 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:13.047857 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:13.047860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:13.048091 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:13.048238 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:13.048253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.049043 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.049048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:13.049572 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:13.049580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.050067 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.050070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:13.050153 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:13.056743 (+ 6590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:13.056755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:13.058790 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":120,"threads_started":1}
W20260810 22:57:13.059079 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:13.047088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46495 (local address 127.29.78.190:46359)
0810 22:57:13.047346 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:13.047350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:13.047474 (+ 124us) server_negotiation.cc:407] Connection header received
0810 22:57:13.047668 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:13.047671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:13.047734 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:13.047830 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:13.048381 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.048937 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.049686 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.049958 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.050061 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:13.058951 (+ 8890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:46495: BlockingRecv error: recv got EOF from 127.29.78.129:46495 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
Aug 10 22:57:13 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:13 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:13.240249 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:13.234620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:60491)
0810 22:57:13.234889 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:13.234903 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:13.235030 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:13.235274 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:13.235276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:13.235491 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:13.235603 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:13.235608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.236398 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.236401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:13.236957 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:13.236965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.237385 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.237388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:13.237451 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:13.238025 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:13.238037 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:13.240094 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
W20260810 22:57:13.240592 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:13.234743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:60491 (local address 127.29.78.190:46359)
0810 22:57:13.234917 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:13.234920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:13.234992 (+ 72us) server_negotiation.cc:407] Connection header received
0810 22:57:13.235115 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:13.235117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:13.235162 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:13.235232 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:13.235702 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.236248 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.237062 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.237270 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.237343 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:13.240328 (+ 2985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:60491: BlockingRecv error: recv got EOF from 127.29.78.130:60491 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 22:57:13 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:13 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:13.422061 7893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:13.416050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:36641)
0810 22:57:13.416359 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:13.416373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:13.416511 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:13.416790 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:13.416793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:13.417099 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:13.417216 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:13.417222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.418094 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.418099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:13.418672 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:13.418680 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.419177 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.419180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:13.419266 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:13.419857 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:13.419870 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:13.421910 (+ 2040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":110,"threads_started":1}
W20260810 22:57:13.422132 7890 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:13.416101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:36641 (local address 127.29.78.190:46359)
0810 22:57:13.416279 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:13.416284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:13.416453 (+ 169us) server_negotiation.cc:407] Connection header received
0810 22:57:13.416625 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:13.416628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:13.416687 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:13.416771 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:13.417355 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.417982 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.418796 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:13.419050 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:13.419157 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:13.422034 (+ 2877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:36641: BlockingRecv error: recv got EOF from 127.29.78.131:36641 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 22:57:14 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:14 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:14.065567 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:14.060053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:39889)
0810 22:57:14.060394 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:14.060411 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:14.060509 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:14.060824 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:14.060826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:14.061054 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:14.061175 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:14.061181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.061965 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.061967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:14.062542 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:14.062551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.062978 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.062980 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:14.063047 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:14.063566 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:14.063578 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:14.065447 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
W20260810 22:57:14.065775 7895 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:14.060159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:39889 (local address 127.29.78.190:46359)
0810 22:57:14.060436 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:14.060440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:14.060478 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:57:14.060606 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:14.060608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:14.060663 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:14.060733 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:14.061282 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.061846 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.062653 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.062873 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.062947 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:14.065593 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:39889: BlockingRecv error: recv got EOF from 127.29.78.129:39889 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":66,"threads_started":1}
Aug 10 22:57:14 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:14 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:14.246934 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:14.241119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:43449)
0810 22:57:14.241447 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:14.241468 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:14.241598 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:14.241926 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:14.241929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:14.242141 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:14.242233 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:14.242238 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.243008 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.243013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:14.243556 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:14.243563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.244041 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.244045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:14.244130 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:14.244617 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:14.244628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:14.246788 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":90,"threads_started":1}
W20260810 22:57:14.247059 7895 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:14.241205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:43449 (local address 127.29.78.190:46359)
0810 22:57:14.241374 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:14.241379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:14.241544 (+ 165us) server_negotiation.cc:407] Connection header received
0810 22:57:14.241720 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:14.241723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:14.241777 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:14.241886 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:14.242351 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.242898 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.243672 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.243927 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.244026 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:14.246925 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:43449: BlockingRecv error: recv got EOF from 127.29.78.130:43449 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 22:57:14 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:14 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:14.428722 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:14.423073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:39345)
0810 22:57:14.423380 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:14.423398 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:14.423511 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:14.423822 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:14.423825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:14.424026 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:14.424126 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:14.424132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.424841 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.424843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:14.425394 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:14.425401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.425840 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.425842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:14.425901 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:14.426414 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:14.426425 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:14.428601 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
W20260810 22:57:14.428870 7895 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:14.423188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:39345 (local address 127.29.78.190:46359)
0810 22:57:14.423330 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:14.423336 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:14.423495 (+ 159us) server_negotiation.cc:407] Connection header received
0810 22:57:14.423618 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:14.423621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:14.423685 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:14.423774 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:14.424232 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.424749 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.425497 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:14.425717 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:14.425791 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:14.428745 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:39345: BlockingRecv error: recv got EOF from 127.29.78.131:39345 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 22:57:15 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:15 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:15.072397 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:15.066518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:52405)
0810 22:57:15.066801 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:15.066815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:15.066910 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:15.067208 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:15.067210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:15.067392 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:15.067505 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:15.067510 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.068289 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.068291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:15.068843 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:15.068850 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.069462 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.069467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:15.069559 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:15.070232 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:15.070243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:15.072277 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":93,"threads_started":1}
W20260810 22:57:15.072662 7900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:15.066598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52405 (local address 127.29.78.190:46359)
0810 22:57:15.066850 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:15.066854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:15.066868 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:57:15.067031 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:15.067034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:15.067085 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:15.067170 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:15.067613 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.068195 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.068979 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.069306 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.069439 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:15.072524 (+ 3085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:52405: BlockingRecv error: recv got EOF from 127.29.78.129:52405 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":92,"threads_started":1}
Aug 10 22:57:15 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:15 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:15.253504 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:15.247655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:40771)
0810 22:57:15.247975 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:15.247988 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:15.248135 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:15.248413 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:15.248416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:15.248642 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:15.248759 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:15.248765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.249560 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.249565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:15.250152 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:15.250160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.250672 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.250675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:15.250763 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:15.251342 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:15.251355 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:15.253335 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
W20260810 22:57:15.253595 7900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:15.247700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:40771 (local address 127.29.78.190:46359)
0810 22:57:15.247883 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:15.247888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:15.248071 (+ 183us) server_negotiation.cc:407] Connection header received
0810 22:57:15.248237 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:15.248240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:15.248295 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:15.248394 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:15.248881 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.249443 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.250276 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.250536 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.250662 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:15.253476 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:40771: BlockingRecv error: recv got EOF from 127.29.78.130:40771 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 22:57:15 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:15 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:15.437340 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:15.429622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:34987)
0810 22:57:15.429984 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:15.430001 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:15.430102 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:15.431993 (+ 1891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:15.431995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:15.432196 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:15.432345 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:15.432355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.433323 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.433326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:15.433919 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:15.433929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.434386 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.434389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:15.434461 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:15.434974 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:15.434986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:15.437224 (+ 2238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":80,"threads_started":1}
W20260810 22:57:15.437518 7900 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:15.429664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:34987 (local address 127.29.78.190:46359)
0810 22:57:15.431720 (+ 2056us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:15.431726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:15.431742 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:57:15.431789 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:15.431793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:15.431848 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:15.431935 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:15.432481 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.433214 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.434023 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:15.434284 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:15.434380 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:15.437360 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:34987: BlockingRecv error: recv got EOF from 127.29.78.131:34987 (error 108)
Metrics: {"server-negotiator.queue_time_us":1960}
Aug 10 22:57:16 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:16 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:16.078545 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:16.073181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:52879)
0810 22:57:16.073473 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:16.073487 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:16.073590 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:16.073849 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:16.073851 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:16.074043 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:16.074147 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:16.074153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.074899 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.074902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:16.075469 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:16.075478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.075892 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.075894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:16.075958 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:16.076500 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:16.076512 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:16.078424 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":88,"threads_started":1}
W20260810 22:57:16.078696 7904 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:16.073251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52879 (local address 127.29.78.190:46359)
0810 22:57:16.073490 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:16.073494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:16.073562 (+ 68us) server_negotiation.cc:407] Connection header received
0810 22:57:16.073670 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:16.073672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:16.073717 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:16.073792 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:16.074248 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.074762 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.075579 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.075791 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.075862 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:16.078555 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:52879: BlockingRecv error: recv got EOF from 127.29.78.129:52879 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
Aug 10 22:57:16 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:16 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:16.260353 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:16.254571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:55833)
0810 22:57:16.254862 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:16.254875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:16.255001 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:16.255265 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:16.255267 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:16.255455 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:16.255551 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:16.255556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.256265 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.256267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:16.256812 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:16.256819 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.257357 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.257363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:16.257446 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:16.258151 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:16.258168 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:16.260237 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
W20260810 22:57:16.260571 7904 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:16.254649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:55833 (local address 127.29.78.190:46359)
0810 22:57:16.254822 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:16.254825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:16.254972 (+ 147us) server_negotiation.cc:407] Connection header received
0810 22:57:16.255106 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:16.255108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:16.255153 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:16.255227 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:16.255651 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.256171 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.256909 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.257198 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.257309 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:16.260396 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:55833: BlockingRecv error: recv got EOF from 127.29.78.130:55833 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 22:57:16 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:16 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:16.444147 7907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:16.438198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:37919)
0810 22:57:16.438568 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:16.438583 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:16.438728 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:16.439015 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:16.439018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:16.439237 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:16.439333 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:16.439339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.440148 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.440153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:16.440809 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:16.440818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.441312 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.441316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:16.441406 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:16.441987 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:16.441998 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:16.443973 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":91,"threads_started":1}
W20260810 22:57:16.444202 7904 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:16.438331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:37919 (local address 127.29.78.190:46359)
0810 22:57:16.438532 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:16.438536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:16.438670 (+ 134us) server_negotiation.cc:407] Connection header received
0810 22:57:16.438840 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:16.438843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:16.438900 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:16.438988 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:16.439495 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.440038 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.440935 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:16.441186 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:16.441295 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:16.444094 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:37919: BlockingRecv error: recv got EOF from 127.29.78.131:37919 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 22:57:17 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:17 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:17.084830 7908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:17.079428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:59133)
0810 22:57:17.079737 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:17.079751 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:17.079866 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:17.080090 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:17.080092 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:17.080289 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:17.080398 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:17.080404 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.081109 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.081112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:17.081685 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:17.081694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.082147 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.082150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:17.082226 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:17.082717 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:17.082729 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:17.084700 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":111,"threads_started":1}
W20260810 22:57:17.084988 7909 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:17.079492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:59133 (local address 127.29.78.190:46359)
0810 22:57:17.079746 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:17.079749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:17.079824 (+ 75us) server_negotiation.cc:407] Connection header received
0810 22:57:17.079950 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:17.079952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:17.079996 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:17.080078 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:17.080495 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.081004 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.081786 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.082034 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.082117 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:17.084838 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:59133: BlockingRecv error: recv got EOF from 127.29.78.129:59133 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":99,"threads_started":1}
Aug 10 22:57:17 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:17 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:17.267047 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:17.261375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:34243)
0810 22:57:17.261719 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:17.261739 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:17.261869 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:17.262149 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:17.262152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:17.262371 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:17.262489 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:17.262494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.263290 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.263295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:17.263860 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:17.263868 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.264349 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.264353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:17.264448 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:17.265037 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:17.265049 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:17.266881 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":128,"threads_started":1}
W20260810 22:57:17.267139 7909 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:17.261657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34243 (local address 127.29.78.190:46359)
0810 22:57:17.261850 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:17.261854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:17.261870 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:57:17.261976 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:17.261979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:17.262030 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:17.262124 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:17.262623 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.263172 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.263998 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.264234 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.264342 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:17.267015 (+ 2673us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:34243: BlockingRecv error: recv got EOF from 127.29.78.130:34243 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 22:57:17 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:17 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:17.451330 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:17.444986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:46369)
0810 22:57:17.445348 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:17.445362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:17.445522 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:17.445854 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:17.445857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:17.446098 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:17.446194 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:17.446201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.447023 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.447045 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:17.447649 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:17.447659 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.448185 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.448189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:17.448284 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:17.448890 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:17.448903 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:17.451147 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":100,"threads_started":1}
W20260810 22:57:17.451388 7909 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:17.445115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:46369 (local address 127.29.78.190:46359)
0810 22:57:17.445286 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:17.445291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:17.445442 (+ 151us) server_negotiation.cc:407] Connection header received
0810 22:57:17.445629 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:17.445634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:17.445694 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:17.445804 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:17.446319 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.446893 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.447781 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:17.448038 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:17.448182 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:17.451276 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:46369: BlockingRecv error: recv got EOF from 127.29.78.131:46369 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 22:57:18 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:18 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:18.096375 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:18.085763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:49753)
0810 22:57:18.086134 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:18.086149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:18.086242 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:18.086535 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:18.086538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:18.086724 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:18.086826 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:18.086831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.087582 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.087588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:18.088385 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:18.088395 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.088926 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.088930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:18.089018 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:18.094313 (+ 5295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:18.094326 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:18.096247 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":137,"threads_started":1}
W20260810 22:57:18.096577 7913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:18.085848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:49753 (local address 127.29.78.190:46359)
0810 22:57:18.086183 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:18.086186 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:18.086200 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:57:18.086324 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:18.086326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:18.086370 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:18.086459 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:18.086936 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.087472 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.088507 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.088801 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.088921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:18.096390 (+ 7469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:49753: BlockingRecv error: recv got EOF from 127.29.78.129:49753 (error 108)
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":109,"threads_started":1}
Aug 10 22:57:18 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:18 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:18.273650 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:18.267980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:41981)
0810 22:57:18.268302 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:18.268316 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:18.268433 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:18.268673 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:18.268676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:18.268879 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:18.268977 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:18.268983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.269728 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.269731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:18.270298 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:18.270306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.270704 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.270707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:18.270769 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:18.271394 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:18.271409 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:18.273517 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":126,"threads_started":1}
W20260810 22:57:18.273936 7913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:18.268018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:41981 (local address 127.29.78.190:46359)
0810 22:57:18.268197 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:18.268202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:18.268409 (+ 207us) server_negotiation.cc:407] Connection header received
0810 22:57:18.268517 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:18.268519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:18.268566 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:18.268626 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:18.269076 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.269614 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.270391 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.270603 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.270677 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:18.273674 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:41981: BlockingRecv error: recv got EOF from 127.29.78.130:41981 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 22:57:18 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:18 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.131@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:18.458122 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:18.452329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.131:56241)
0810 22:57:18.452679 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:18.452692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:18.452846 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:18.453114 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:18.453116 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:18.453323 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:18.453439 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:18.453444 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.454246 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.454250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:18.454784 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:18.454791 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.455237 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.455240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:18.455318 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:18.455928 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:18.455938 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:18.457922 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
W20260810 22:57:18.458243 7913 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:18.452403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:56241 (local address 127.29.78.190:46359)
0810 22:57:18.452617 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:18.452622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:18.452772 (+ 150us) server_negotiation.cc:407] Connection header received
0810 22:57:18.452952 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:18.452957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:18.453008 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:18.453088 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:18.453555 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.454142 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.454900 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:18.455119 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:18.455232 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:18.458073 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.131:56241: BlockingRecv error: recv got EOF from 127.29.78.131:56241 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 10 22:57:19 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:19 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.129@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:19.102890 7917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:19.097272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.129:47781)
0810 22:57:19.097530 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:19.097550 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:19.097642 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:19.097938 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:19.097940 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:19.098177 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:19.098305 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:19.098330 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.099104 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.099107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:19.099744 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:19.099752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.100148 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.100150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:19.100210 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:19.100693 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:19.100704 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:19.102780 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":68,"threads_started":1}
W20260810 22:57:19.103046 7918 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:19.097368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:47781 (local address 127.29.78.190:46359)
0810 22:57:19.097608 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:19.097611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:19.097623 (+ 12us) server_negotiation.cc:407] Connection header received
0810 22:57:19.097723 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:19.097725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:19.097765 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:19.097850 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:19.098444 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.099001 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.099857 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.100058 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.100119 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:19.102908 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.129:47781: BlockingRecv error: recv got EOF from 127.29.78.129:47781 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
Aug 10 22:57:19 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 22:57:19 dist-test-slave-6nmv krb5kdc[6708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.29.78.130@KRBTEST.COM for oryx/127.29.78.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 22:57:19.279665 7919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:19.274594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:46359 (local address 127.29.78.130:59585)
0810 22:57:19.274864 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:19.274879 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:19.274974 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:19.275161 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:19.275162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:19.275348 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:19.275450 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:19.275455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.276199 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.276202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:19.276733 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:19.276741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.277145 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.277148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:19.277206 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:19.277668 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:19.277679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:19.279534 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.29.78.190:46359: Server oryx/127.29.78.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":101,"threads_started":1}
W20260810 22:57:19.279881 7918 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:19.274651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:59585 (local address 127.29.78.190:46359)
0810 22:57:19.274806 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:19.274809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:19.274946 (+ 137us) server_negotiation.cc:407] Connection header received
0810 22:57:19.275040 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:19.275042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:19.275083 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:19.275148 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:19.275553 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.276069 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.276837 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.277051 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.277126 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:19.279684 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.29.78.130:59585: BlockingRecv error: recv got EOF from 127.29.78.130:59585 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:57:19.351226 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 7376
I20260810 22:57:19.358685 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 7543
I20260810 22:57:19.366392 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 7690
I20260810 22:57:19.373950 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 7299
2026-08-10T22:57:19Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18025 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T22:57:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:57:19Z Disabled control of system clock
I20260810 22:57:19.405857 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40683
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 22:57:19.510443 7925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:19.510672 7925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:19.510701 7925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:19.514359 7925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:57:19.514401 7925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:19.514415 7925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:19.514429 7925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:57:19.514467 7925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:57:19.518890 7925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:19.519922 7925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:19.521263 7925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:19.529561 7930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:19.529551 7931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:19.529599 7925 server_base.cc:1034] running on GCE node
W20260810 22:57:19.529541 7935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:19.530247 7925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:19.531272 7925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:19.532474 7925 hybrid_clock.cc:648] HybridClock initialized: now 1786402639532457 us; error 30 us; skew 500 ppm
I20260810 22:57:19.534264 7925 webserver.cc:460] Webserver started at http://127.29.78.190:43867/ using document root <none> and password file <none>
I20260810 22:57:19.534801 7925 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:19.534896 7925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:19.535115 7925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:19.536962 7925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "a0102bddb45148db9b7accaf4ea7df6b"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "eb74fabbc7f0269a32605f96a3e187db"
server_key_iv: "4fc35bfa02ccc93098bd3a5b71d27b85"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.537534 7925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "a0102bddb45148db9b7accaf4ea7df6b"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "eb74fabbc7f0269a32605f96a3e187db"
server_key_iv: "4fc35bfa02ccc93098bd3a5b71d27b85"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.541178 7925 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:57:19.543502 7941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:19.544679 7925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 22:57:19.544816 7925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "a0102bddb45148db9b7accaf4ea7df6b"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "eb74fabbc7f0269a32605f96a3e187db"
server_key_iv: "4fc35bfa02ccc93098bd3a5b71d27b85"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.544922 7925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:19.567881 7925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:19.568571 7925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:19.568778 7925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:19.575861 7925 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40683
I20260810 22:57:19.575888 8000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40683 every 8 connection(s)
I20260810 22:57:19.576758 7925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:57:19.579890 8001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:57:19.581153 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 7925
I20260810 22:57:19.581296 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:57:19.581676 30010 external_mini_cluster.cc:1428] Setting key c15ed091edda0cb0184a75bc89cbadf1
I20260810 22:57:19.585680 8001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b: Bootstrap starting.
I20260810 22:57:19.587724 8001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b: Neither blocks nor log segments found. Creating new log.
I20260810 22:57:19.588483 8001 log.cc:824] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:19.590323 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:19.583075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50494 (local address 127.29.78.190:40683)
0810 22:57:19.583640 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:19.583652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:19.583683 (+ 31us) server_negotiation.cc:407] Connection header received
0810 22:57:19.584267 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:19.584281 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:19.584627 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:19.584956 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:19.585332 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.586267 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.587081 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.587532 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.587697 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:19.588611 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:19.588619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:19.588637 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:19.589022 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:19.589075 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:19.589095 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:19.589207 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:19.589323 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 22:57:19.589591 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":118,"threads_started":1}
I20260810 22:57:19.590431 8001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b: No bootstrap required, opened a new log
I20260810 22:57:19.594103 8001 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0102bddb45148db9b7accaf4ea7df6b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40683 } }
I20260810 22:57:19.594344 8001 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:57:19.594420 8001 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0102bddb45148db9b7accaf4ea7df6b, State: Initialized, Role: FOLLOWER
I20260810 22:57:19.594805 8001 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0102bddb45148db9b7accaf4ea7df6b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40683 } }
I20260810 22:57:19.594960 8001 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:57:19.595044 8001 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:57:19.595170 8001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:57:19.596030 8001 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0102bddb45148db9b7accaf4ea7df6b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40683 } }
I20260810 22:57:19.596330 8001 leader_election.cc:302] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0102bddb45148db9b7accaf4ea7df6b; no voters:
I20260810 22:57:19.596603 8001 leader_election.cc:288] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:57:19.596712 8008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:57:19.596944 8008 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [term 1 LEADER]: Becoming Leader. State: Replica: a0102bddb45148db9b7accaf4ea7df6b, State: Running, Role: LEADER
I20260810 22:57:19.597289 8008 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0102bddb45148db9b7accaf4ea7df6b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40683 } }
I20260810 22:57:19.597718 8001 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:57:19.598860 8009 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0102bddb45148db9b7accaf4ea7df6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0102bddb45148db9b7accaf4ea7df6b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40683 } } }
I20260810 22:57:19.598994 8009 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:19.599169 8010 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0102bddb45148db9b7accaf4ea7df6b. Latest consensus state: current_term: 1 leader_uuid: "a0102bddb45148db9b7accaf4ea7df6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0102bddb45148db9b7accaf4ea7df6b" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40683 } } }
I20260810 22:57:19.599334 8010 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:19.599622 8020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:57:19.602936 8020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:57:19.608078 8020 catalog_manager.cc:1347] Generated new cluster ID: 703220c4c7e74674a80fbc23ebfa9ef9
I20260810 22:57:19.608155 8020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:57:19.626021 8020 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:57:19.627274 8020 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:57:19.634490 8020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0102bddb45148db9b7accaf4ea7df6b: Generated new TSK 0
I20260810 22:57:19.635233 8020 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:57:19.638549 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40683
--builtin_ntp_servers=127.29.78.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 22:57:19.746924 8028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:19.747208 8028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:19.747248 8028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:19.751415 8028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:19.751461 8028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:19.751510 8028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:57:19.756054 8028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:19.757244 8028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:19.758632 8028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 22:57:19.768505 8028 server_base.cc:1034] running on GCE node
W20260810 22:57:19.768446 8034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:19.768447 8033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:19.768486 8036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:19.769119 8028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:19.769738 8028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:19.770931 8028 hybrid_clock.cc:648] HybridClock initialized: now 1786402639770919 us; error 24 us; skew 500 ppm
I20260810 22:57:19.772743 8028 webserver.cc:460] Webserver started at http://127.29.78.129:33811/ using document root <none> and password file <none>
I20260810 22:57:19.773319 8028 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:19.773412 8028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:19.773653 8028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:19.775796 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "eb26d7e98f1743618026f04469cb0256"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "1dff4823ae8e0533e67d7aa4c296cc19"
server_key_iv: "db0f7b1eca03d18bf13d82509890dda2"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.776373 8028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "eb26d7e98f1743618026f04469cb0256"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "1dff4823ae8e0533e67d7aa4c296cc19"
server_key_iv: "db0f7b1eca03d18bf13d82509890dda2"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.780043 8028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 22:57:19.782667 8042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:19.783890 8028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:57:19.784065 8028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "eb26d7e98f1743618026f04469cb0256"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "1dff4823ae8e0533e67d7aa4c296cc19"
server_key_iv: "db0f7b1eca03d18bf13d82509890dda2"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.784192 8028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:19.834908 8028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:19.835623 8028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:19.835821 8028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:19.836401 8028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:19.837483 8028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:19.837543 8028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:19.837587 8028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:19.837604 8028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:19.846510 8028 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:32827
I20260810 22:57:19.846534 8168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:32827 every 8 connection(s)
I20260810 22:57:19.847379 8028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:57:19.855875 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 8028
I20260810 22:57:19.856001 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:57:19.856348 30010 external_mini_cluster.cc:1428] Setting key 37d5620984a42f19cc57508ee8bce633
I20260810 22:57:19.856540 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:19.849298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:47209 (local address 127.29.78.190:40683)
0810 22:57:19.849437 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:19.849448 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:19.850480 (+ 1032us) server_negotiation.cc:407] Connection header received
0810 22:57:19.851791 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:19.851794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:19.851860 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:19.851946 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:19.852791 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.853290 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.854504 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.854739 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.854856 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:19.856020 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:19.856024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:19.856026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:19.856053 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:19.856079 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:19.856095 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:19.856155 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:19.856315 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 22:57:19.856369 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:57:19.857708 8171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:19.849689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40683 (local address 127.29.78.129:47209)
0810 22:57:19.850343 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:19.850383 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:19.851562 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:19.852112 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:19.852118 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:19.852195 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:19.852650 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:19.852662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.853409 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.853418 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:19.854370 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:19.854380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:19.854905 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:19.854909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:19.855057 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:19.855689 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:57:19.855693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:19.855742 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:57:19.855819 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:19.856164 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:19.856176 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:19.856182 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:57:19.856307 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 22:57:19.856570 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":162,"threads_started":1}
I20260810 22:57:19.858568 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40683
--builtin_ntp_servers=127.29.78.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 22:57:19.858951 8169 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40683
I20260810 22:57:19.859190 8169 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:19.859683 8169 heartbeater.cc:511] Master 127.29.78.190:40683 requested a full tablet report, sending...
I20260810 22:57:19.861038 7960 ts_manager.cc:194] Registered new tserver with Master: eb26d7e98f1743618026f04469cb0256 (127.29.78.129:32827)
I20260810 22:57:19.862241 7960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:47209
W20260810 22:57:19.966306 8172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:19.966519 8172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:19.966588 8172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:19.970418 8172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:19.970461 8172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:19.970510 8172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:57:19.975044 8172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:19.976029 8172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:19.977305 8172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:19.986629 8177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:19.986744 8178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:19.986902 8172 server_base.cc:1034] running on GCE node
W20260810 22:57:19.987083 8180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:19.987488 8172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:19.988128 8172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:19.989301 8172 hybrid_clock.cc:648] HybridClock initialized: now 1786402639989284 us; error 33 us; skew 500 ppm
I20260810 22:57:19.991041 8172 webserver.cc:460] Webserver started at http://127.29.78.130:34715/ using document root <none> and password file <none>
I20260810 22:57:19.991606 8172 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:19.991719 8172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:19.991950 8172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:19.993939 8172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "e214a126a27c4a4bb6ef3f1d696c6e0e"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "ad4b52c7f4bc99b6652bb95a05c3c2bf"
server_key_iv: "4f73372b6701a5a5923db9fd4fbcbf6a"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.994501 8172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "e214a126a27c4a4bb6ef3f1d696c6e0e"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "ad4b52c7f4bc99b6652bb95a05c3c2bf"
server_key_iv: "4f73372b6701a5a5923db9fd4fbcbf6a"
server_key_version: "encryptionkey@0"
I20260810 22:57:19.997890 8172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:57:20.000295 8186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.001545 8172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:57:20.001660 8172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "e214a126a27c4a4bb6ef3f1d696c6e0e"
format_stamp: "Formatted at 2026-08-10 22:57:19 on dist-test-slave-6nmv"
server_key: "ad4b52c7f4bc99b6652bb95a05c3c2bf"
server_key_iv: "4f73372b6701a5a5923db9fd4fbcbf6a"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.001745 8172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:20.016825 8172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:20.017522 8172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:20.017681 8172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:20.018340 8172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:20.019308 8172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:20.019357 8172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.019397 8172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:20.019423 8172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.028815 8172 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:34037
I20260810 22:57:20.028846 8316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:34037 every 8 connection(s)
I20260810 22:57:20.029718 8172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:57:20.035418 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 8172
I20260810 22:57:20.035547 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:57:20.035864 30010 external_mini_cluster.cc:1428] Setting key 876178edde96b39c4f0193702fe9e895
I20260810 22:57:20.037750 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40683
--builtin_ntp_servers=127.29.78.148:45181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 22:57:20.039585 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.031584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:34241 (local address 127.29.78.190:40683)
0810 22:57:20.031743 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:20.031746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:20.032799 (+ 1053us) server_negotiation.cc:407] Connection header received
0810 22:57:20.034298 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:20.034301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:20.034363 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:20.034468 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:20.035379 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.036040 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.037411 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.037640 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.037754 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:20.038987 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:20.038990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:20.038993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:20.039006 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:20.039036 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:20.039045 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:20.039127 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:20.039291 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 22:57:20.039441 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:57:20.040436 8320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.031993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40683 (local address 127.29.78.130:34241)
0810 22:57:20.032652 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:20.032693 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:20.034082 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:20.034647 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:20.034653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:20.034732 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:20.035226 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:20.035239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.036171 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.036179 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:20.037285 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:20.037296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.037796 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.037800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:20.038051 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:20.038705 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:57:20.038706 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:20.038745 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:57:20.038825 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:20.039129 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:20.039141 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:20.039150 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 22:57:20.039286 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 22:57:20.039672 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":125,"threads_started":1}
I20260810 22:57:20.041604 8317 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40683
I20260810 22:57:20.041895 8317 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:20.042455 8317 heartbeater.cc:511] Master 127.29.78.190:40683 requested a full tablet report, sending...
I20260810 22:57:20.043460 7960 ts_manager.cc:194] Registered new tserver with Master: e214a126a27c4a4bb6ef3f1d696c6e0e (127.29.78.130:34037)
I20260810 22:57:20.044018 7960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:34241
W20260810 22:57:20.148885 8321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:20.149159 8321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:20.149235 8321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:20.153263 8321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:20.153334 8321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:20.153414 8321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:57:20.158298 8321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:20.159271 8321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:20.160566 8321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:20.169708 8326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:20.169716 8327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.169745 8321 server_base.cc:1034] running on GCE node
W20260810 22:57:20.169715 8329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.170258 8321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:20.170853 8321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:20.172034 8321 hybrid_clock.cc:648] HybridClock initialized: now 1786402640172006 us; error 37 us; skew 500 ppm
I20260810 22:57:20.173920 8321 webserver.cc:460] Webserver started at http://127.29.78.131:41961/ using document root <none> and password file <none>
I20260810 22:57:20.174506 8321 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:20.174589 8321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:20.174786 8321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:20.176653 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "d9cb1874d7aa4e7e83270a44ba45d61b"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "e73a568fdde6ed1272b4992e68059c83"
server_key_iv: "39ade80e520e4f8a8a3531d9a3123c8c"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.177189 8321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "d9cb1874d7aa4e7e83270a44ba45d61b"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "e73a568fdde6ed1272b4992e68059c83"
server_key_iv: "39ade80e520e4f8a8a3531d9a3123c8c"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.180740 8321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:57:20.183120 8336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.184231 8321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:57:20.184316 8321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "d9cb1874d7aa4e7e83270a44ba45d61b"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "e73a568fdde6ed1272b4992e68059c83"
server_key_iv: "39ade80e520e4f8a8a3531d9a3123c8c"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.184383 8321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:20.195638 8321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:20.196134 8321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:20.196271 8321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:20.196731 8321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:20.197500 8321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:20.197559 8321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.197592 8321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:20.197611 8321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.206167 8321 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:46507
I20260810 22:57:20.206187 8463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:46507 every 8 connection(s)
I20260810 22:57:20.207015 8321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:57:20.214725 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 8321
I20260810 22:57:20.214849 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:57:20.215126 30010 external_mini_cluster.cc:1428] Setting key cd107ca5f7ccc738589eb304422fb6a9
I20260810 22:57:20.215919 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.208767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:43425 (local address 127.29.78.190:40683)
0810 22:57:20.208905 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:20.208908 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:20.209983 (+ 1075us) server_negotiation.cc:407] Connection header received
0810 22:57:20.211394 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:20.211397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:20.211452 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:20.211538 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:20.212391 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.212920 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.214005 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.214232 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.214328 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:20.215454 (+ 1126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:20.215457 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:20.215460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:20.215472 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:20.215494 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:20.215510 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:20.215583 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:20.215735 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 22:57:20.215792 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:57:20.216656 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.209156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40683 (local address 127.29.78.131:43425)
0810 22:57:20.209787 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:20.209882 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:20.211199 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:20.211728 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:20.211735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:20.211808 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:20.212243 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:20.212255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.213046 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.213053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:20.213893 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:20.213903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:20.214373 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:20.214394 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:20.214552 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:20.215181 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:57:20.215182 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:20.215243 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:57:20.215314 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:20.215585 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:20.215596 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:20.215603 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 22:57:20.215729 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 22:57:20.215968 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260810 22:57:20.217600 8464 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40683
I20260810 22:57:20.217770 8464 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:20.218231 8464 heartbeater.cc:511] Master 127.29.78.190:40683 requested a full tablet report, sending...
I20260810 22:57:20.219166 7960 ts_manager.cc:194] Registered new tserver with Master: d9cb1874d7aa4e7e83270a44ba45d61b (127.29.78.131:46507)
I20260810 22:57:20.219674 7960 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:43425
I20260810 22:57:20.228968 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:57:20.231747 8005 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:20.230872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.29.78.190:40683)
0810 22:57:20.231015 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:20.231018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:20.231173 (+ 155us) server_negotiation.cc:407] Connection header received
0810 22:57:20.231267 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:20.231269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:20.231304 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:20.231370 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:20.231569 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50496: BlockingRecv error: recv got EOF from 127.0.0.1:50496 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 22:57:20.232192 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 8028
I20260810 22:57:20.239825 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 8172
I20260810 22:57:20.248309 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 8321
I20260810 22:57:20.255824 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 7925
2026-08-10T22:57:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (879 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T22:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:57:20Z Disabled control of system clock
I20260810 22:57:20.285102 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38279
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:38279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:57:20.390549 8475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:20.390782 8475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:20.390818 8475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:20.394526 8475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:57:20.394567 8475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:20.394582 8475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:20.394596 8475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:57:20.394635 8475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:57:20.398939 8475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:38279
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:20.399960 8475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:20.401183 8475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:20.409070 8480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:20.409078 8481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.409116 8475 server_base.cc:1034] running on GCE node
W20260810 22:57:20.409080 8483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.409940 8475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:20.410952 8475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:20.412117 8475 hybrid_clock.cc:648] HybridClock initialized: now 1786402640412096 us; error 34 us; skew 500 ppm
I20260810 22:57:20.413959 8475 webserver.cc:460] Webserver started at http://127.29.78.190:44653/ using document root <none> and password file <none>
I20260810 22:57:20.414422 8475 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:20.414472 8475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:20.414628 8475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:20.416482 8475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "cde8f3dbb9474f649657677d1c7e9002"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "5298c313dbed3ba505d87b5c1781f658"
server_key_iv: "b81f65c3e4aa32aa4a31d0cbcaa61e42"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.417042 8475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "cde8f3dbb9474f649657677d1c7e9002"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "5298c313dbed3ba505d87b5c1781f658"
server_key_iv: "b81f65c3e4aa32aa4a31d0cbcaa61e42"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.420475 8475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 22:57:20.422644 8491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.423799 8475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:57:20.423892 8475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "cde8f3dbb9474f649657677d1c7e9002"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "5298c313dbed3ba505d87b5c1781f658"
server_key_iv: "b81f65c3e4aa32aa4a31d0cbcaa61e42"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.423967 8475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:20.427966 8475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:20.428491 8475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:20.428629 8475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:20.435333 8475 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:38279
I20260810 22:57:20.435339 8550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:38279 every 8 connection(s)
I20260810 22:57:20.436156 8475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:57:20.439203 8551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:57:20.440510 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 8475
I20260810 22:57:20.440590 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:57:20.440850 30010 external_mini_cluster.cc:1428] Setting key 78b2e939f1c7118f2ff251763dabdc72
I20260810 22:57:20.445485 8551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002: Bootstrap starting.
I20260810 22:57:20.446529 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.442203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.29.78.190:38279)
0810 22:57:20.442798 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:20.442817 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:20.442855 (+ 38us) server_negotiation.cc:407] Connection header received
0810 22:57:20.443650 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:20.443670 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:20.444081 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:20.444443 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:20.445146 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:20.445152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:20.445177 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:20.445187 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:20.445249 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:20.445268 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:20.445396 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:20.445470 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 22:57:20.445710 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
I20260810 22:57:20.447927 8551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002: Neither blocks nor log segments found. Creating new log.
I20260810 22:57:20.448735 8551 log.cc:824] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:20.450570 8551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002: No bootstrap required, opened a new log
I20260810 22:57:20.455962 8551 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8f3dbb9474f649657677d1c7e9002" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38279 } }
I20260810 22:57:20.456229 8551 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:57:20.456301 8551 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde8f3dbb9474f649657677d1c7e9002, State: Initialized, Role: FOLLOWER
I20260810 22:57:20.456794 8551 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8f3dbb9474f649657677d1c7e9002" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38279 } }
I20260810 22:57:20.456959 8551 raft_consensus.cc:388] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:57:20.457024 8551 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:57:20.457118 8551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:57:20.458446 8551 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8f3dbb9474f649657677d1c7e9002" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38279 } }
I20260810 22:57:20.458819 8551 leader_election.cc:302] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cde8f3dbb9474f649657677d1c7e9002; no voters:
I20260810 22:57:20.459177 8551 leader_election.cc:288] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:57:20.459291 8557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:57:20.459492 8557 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [term 1 LEADER]: Becoming Leader. State: Replica: cde8f3dbb9474f649657677d1c7e9002, State: Running, Role: LEADER
I20260810 22:57:20.459895 8557 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8f3dbb9474f649657677d1c7e9002" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38279 } }
I20260810 22:57:20.460404 8551 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:57:20.461076 8560 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde8f3dbb9474f649657677d1c7e9002. Latest consensus state: current_term: 1 leader_uuid: "cde8f3dbb9474f649657677d1c7e9002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8f3dbb9474f649657677d1c7e9002" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38279 } } }
I20260810 22:57:20.461272 8560 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:20.461519 8559 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cde8f3dbb9474f649657677d1c7e9002" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cde8f3dbb9474f649657677d1c7e9002" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38279 } } }
I20260810 22:57:20.461606 8559 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002 [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:20.461746 8568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:57:20.464337 8568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:57:20.469625 8568 catalog_manager.cc:1347] Generated new cluster ID: 89f776e717334017b75a398a21e4497b
I20260810 22:57:20.469697 8568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:57:20.500247 8568 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:57:20.500598 8568 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:57:20.508822 8568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cde8f3dbb9474f649657677d1c7e9002: Generated new TSK 0
I20260810 22:57:20.509467 8568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:57:20.515321 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38279
--builtin_ntp_servers=127.29.78.148:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:57:20.635179 8581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:20.635422 8581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:20.635461 8581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:20.639123 8581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:20.639174 8581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:20.639230 8581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:57:20.643299 8581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:20.644250 8581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:20.645493 8581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:20.654863 8589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:20.654886 8586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:20.654903 8587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.654937 8581 server_base.cc:1034] running on GCE node
I20260810 22:57:20.655506 8581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:20.656102 8581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:20.657286 8581 hybrid_clock.cc:648] HybridClock initialized: now 1786402640657278 us; error 27 us; skew 500 ppm
I20260810 22:57:20.659054 8581 webserver.cc:460] Webserver started at http://127.29.78.129:46659/ using document root <none> and password file <none>
I20260810 22:57:20.659627 8581 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:20.659684 8581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:20.659919 8581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:20.661936 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "44042f4a906740d79cab8c58b45a91a0"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "b162301aa29964256941c0c484eee302"
server_key_iv: "d3caeea01eec694472a83cd4005d7434"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.662498 8581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "44042f4a906740d79cab8c58b45a91a0"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "b162301aa29964256941c0c484eee302"
server_key_iv: "d3caeea01eec694472a83cd4005d7434"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.666029 8581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:57:20.668254 8597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.669373 8581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:57:20.669512 8581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "44042f4a906740d79cab8c58b45a91a0"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "b162301aa29964256941c0c484eee302"
server_key_iv: "d3caeea01eec694472a83cd4005d7434"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.669627 8581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:20.673904 8581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:20.674469 8581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:20.674654 8581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:20.675163 8581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:20.676002 8581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:20.676070 8581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.676124 8581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:20.676172 8581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.684736 8581 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:39757
I20260810 22:57:20.684760 8722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:39757 every 8 connection(s)
I20260810 22:57:20.685561 8581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:57:20.691030 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.687227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:44031 (local address 127.29.78.190:38279)
0810 22:57:20.687354 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:20.687358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:20.688314 (+ 956us) server_negotiation.cc:407] Connection header received
0810 22:57:20.689528 (+ 1214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:20.689531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:20.689580 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:20.689662 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:20.690692 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:20.690694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:20.690696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:20.690705 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:20.690721 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:20.690728 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:20.690794 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:20.690896 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 22:57:20.690935 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:57:20.691761 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.687594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38279 (local address 127.29.78.129:44031)
0810 22:57:20.688187 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:20.688224 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:20.689356 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:20.689801 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:20.689841 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:20.689932 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:20.690477 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:57:20.690478 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:20.690514 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:57:20.690571 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:20.690796 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:20.690800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:20.690803 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 22:57:20.690908 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 22:57:20.691072 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":106,"threads_started":1}
I20260810 22:57:20.692910 8723 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38279
I20260810 22:57:20.693118 8723 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:20.693468 8723 heartbeater.cc:511] Master 127.29.78.190:38279 requested a full tablet report, sending...
I20260810 22:57:20.694793 8511 ts_manager.cc:194] Registered new tserver with Master: 44042f4a906740d79cab8c58b45a91a0 (127.29.78.129:39757)
I20260810 22:57:20.695279 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 8581
I20260810 22:57:20.695402 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:57:20.695722 30010 external_mini_cluster.cc:1428] Setting key 9b481a3088b34e0f436beaeeaec4c928
I20260810 22:57:20.697465 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38279
--builtin_ntp_servers=127.29.78.148:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 22:57:20.800127 8727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:20.800338 8727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:20.800390 8727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:20.803730 8727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:20.803767 8727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:20.803814 8727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:57:20.807780 8727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:20.808652 8727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:20.809902 8727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:20.818477 8732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:20.818466 8733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.818570 8727 server_base.cc:1034] running on GCE node
W20260810 22:57:20.818499 8737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.819036 8727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:20.819667 8727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:20.820856 8727 hybrid_clock.cc:648] HybridClock initialized: now 1786402640820824 us; error 44 us; skew 500 ppm
I20260810 22:57:20.822664 8727 webserver.cc:460] Webserver started at http://127.29.78.130:46401/ using document root <none> and password file <none>
I20260810 22:57:20.823256 8727 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:20.823346 8727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:20.823577 8727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:20.825531 8727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "d70b41ebbca840aba9581126d2bc7682"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "4b8f14790b62870758d2891eec513ede"
server_key_iv: "12664dfc0a4478c5a8daad7c4e6b312a"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.826095 8727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "d70b41ebbca840aba9581126d2bc7682"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "4b8f14790b62870758d2891eec513ede"
server_key_iv: "12664dfc0a4478c5a8daad7c4e6b312a"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.829975 8727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 22:57:20.832342 8743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.833621 8727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:57:20.833757 8727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "d70b41ebbca840aba9581126d2bc7682"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "4b8f14790b62870758d2891eec513ede"
server_key_iv: "12664dfc0a4478c5a8daad7c4e6b312a"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.833953 8727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:20.838127 8727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:20.838732 8727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:20.838917 8727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:20.839438 8727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:20.840323 8727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:20.840394 8727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.840452 8727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:20.840512 8727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.848542 8727 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:36119
I20260810 22:57:20.848608 8871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:36119 every 8 connection(s)
I20260810 22:57:20.849385 8727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:57:20.853543 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 8727
I20260810 22:57:20.853698 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:57:20.854040 30010 external_mini_cluster.cc:1428] Setting key 61a53e532148ad2d72f8a334c67b14f4
I20260810 22:57:20.855475 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.851130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:58215 (local address 127.29.78.190:38279)
0810 22:57:20.851282 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:20.851286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:20.852275 (+ 989us) server_negotiation.cc:407] Connection header received
0810 22:57:20.853523 (+ 1248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:20.853526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:20.853561 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:20.853772 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:20.855047 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:20.855050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:20.855053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:20.855063 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:20.855088 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:20.855097 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:20.855179 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:20.855297 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 22:57:20.855338 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:57:20.856389 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38279
--builtin_ntp_servers=127.29.78.148:35657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 22:57:20.856406 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:20.851508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38279 (local address 127.29.78.130:58215)
0810 22:57:20.852122 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:20.852159 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:20.853299 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:20.853979 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:20.853987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:20.854083 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:20.854779 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:57:20.854781 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:20.854829 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:57:20.854908 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:20.855175 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:20.855181 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:20.855185 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 22:57:20.855299 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 22:57:20.855530 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":103,"threads_started":1}
I20260810 22:57:20.857492 8872 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38279
I20260810 22:57:20.857703 8872 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:20.858192 8872 heartbeater.cc:511] Master 127.29.78.190:38279 requested a full tablet report, sending...
I20260810 22:57:20.859278 8511 ts_manager.cc:194] Registered new tserver with Master: d70b41ebbca840aba9581126d2bc7682 (127.29.78.130:36119)
W20260810 22:57:20.962986 8876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:20.963218 8876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:20.963277 8876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:20.966959 8876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:20.967028 8876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:20.967104 8876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:57:20.971377 8876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:20.972471 8876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:20.973800 8876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:20.982577 8886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:20.982584 8882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:20.982590 8883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:20.982944 8876 server_base.cc:1034] running on GCE node
I20260810 22:57:20.983309 8876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:20.983891 8876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:20.985095 8876 hybrid_clock.cc:648] HybridClock initialized: now 1786402640985079 us; error 32 us; skew 500 ppm
I20260810 22:57:20.986865 8876 webserver.cc:460] Webserver started at http://127.29.78.131:43507/ using document root <none> and password file <none>
I20260810 22:57:20.987442 8876 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:20.987526 8876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:20.987739 8876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:20.989686 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "4050f1eac8524788940691cfeb8975e9"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "d0a5db7515dae5772b9a34a09aecb890"
server_key_iv: "14b31d35b845dddf89b626939f8996b9"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.990331 8876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "4050f1eac8524788940691cfeb8975e9"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "d0a5db7515dae5772b9a34a09aecb890"
server_key_iv: "14b31d35b845dddf89b626939f8996b9"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.993784 8876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:57:20.996157 8893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:20.997300 8876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:57:20.997426 8876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "4050f1eac8524788940691cfeb8975e9"
format_stamp: "Formatted at 2026-08-10 22:57:20 on dist-test-slave-6nmv"
server_key: "d0a5db7515dae5772b9a34a09aecb890"
server_key_iv: "14b31d35b845dddf89b626939f8996b9"
server_key_version: "encryptionkey@0"
I20260810 22:57:20.997530 8876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:21.001886 8876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:21.002473 8876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:21.002651 8876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:21.003165 8876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:21.004022 8876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:21.004067 8876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:21.004101 8876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:21.004117 8876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:21.014469 8876 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:34393
I20260810 22:57:21.014560 9021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:34393 every 8 connection(s)
I20260810 22:57:21.015367 8876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:57:21.021692 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:21.017459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:47413 (local address 127.29.78.190:38279)
0810 22:57:21.017598 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:21.017601 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:21.018675 (+ 1074us) server_negotiation.cc:407] Connection header received
0810 22:57:21.020006 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:21.020009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:21.020053 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:21.020152 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:21.021214 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:21.021217 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:21.021220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 22:57:21.021233 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:21.021255 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 22:57:21.021270 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:21.021364 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:21.021519 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 22:57:21.021566 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:57:21.022470 9024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:21.017794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38279 (local address 127.29.78.131:47413)
0810 22:57:21.018534 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:21.018572 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:21.019783 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:21.020304 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:21.020310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:21.020390 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:21.020955 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 22:57:21.020957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:21.020994 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 22:57:21.021065 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:21.021350 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:21.021358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:21.021364 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 22:57:21.021543 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 22:57:21.021760 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260810 22:57:21.023464 9022 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38279
I20260810 22:57:21.023681 9022 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:21.024219 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 8876
I20260810 22:57:21.024235 9022 heartbeater.cc:511] Master 127.29.78.190:38279 requested a full tablet report, sending...
I20260810 22:57:21.024458 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:57:21.024745 30010 external_mini_cluster.cc:1428] Setting key fa8ff15f3ff0cf5d01b01e8ab0c692ba
I20260810 22:57:21.025528 8511 ts_manager.cc:194] Registered new tserver with Master: 4050f1eac8524788940691cfeb8975e9 (127.29.78.131:34393)
I20260810 22:57:21.027096 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 22:57:21.030079 8555 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:57:21.029146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.29.78.190:38279)
0810 22:57:21.029292 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:21.029297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:21.029495 (+ 198us) server_negotiation.cc:407] Connection header received
0810 22:57:21.029570 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:21.029572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:21.029605 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:21.029687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:21.029963 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50202: BlockingRecv error: recv got EOF from 127.0.0.1:50202 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 22:57:21.030198 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 8581
I20260810 22:57:21.043198 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 8727
I20260810 22:57:21.049609 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 8876
I20260810 22:57:21.055888 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 8475
2026-08-10T22:57:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (802 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:57:21 dist-test-slave-6nmv krb5kdc[9034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:57:21 dist-test-slave-6nmv krb5kdc[9034](info): set up 2 sockets
Aug 10 22:57:21 dist-test-slave-6nmv krb5kdc[9034](info): commencing operation
krb5kdc: starting...
W20260810 22:57:22.102732 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:57:22 dist-test-slave-6nmv krb5kdc[9034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:57:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:57:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:57:22.228770 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43743
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:22.334739 9048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:22.334978 9048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:22.335013 9048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:22.338656 9048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:57:22.338739 9048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:22.338776 9048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:22.338793 9048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:57:22.338809 9048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:57:22.343567 9048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:22.344504 9048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:22.345400 9048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:22.353647 9054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:22.353694 9055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:22.353837 9048 server_base.cc:1034] running on GCE node
W20260810 22:57:22.353663 9057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:22.354481 9048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:22.355525 9048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:22.356714 9048 hybrid_clock.cc:648] HybridClock initialized: now 1786402642356685 us; error 40 us; skew 500 ppm
Aug 10 22:57:22 dist-test-slave-6nmv krb5kdc[9034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402642, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:22.360033 9048 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:57:22.360918 9048 webserver.cc:460] Webserver started at http://127.29.78.190:40249/ using document root <none> and password file <none>
I20260810 22:57:22.361451 9048 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:22.361557 9048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:22.361783 9048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:22.363813 9048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "40e310befbf24f81ac694043f5e57232"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "81073ea5793b9fe3f05d2a0af0441bb0"
server_key_iv: "3daa55436cc4dc9d6b7c6d8d9da0b2b3"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.364382 9048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "40e310befbf24f81ac694043f5e57232"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "81073ea5793b9fe3f05d2a0af0441bb0"
server_key_iv: "3daa55436cc4dc9d6b7c6d8d9da0b2b3"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.367944 9048 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 22:57:22.370340 9066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:22.371538 9048 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:57:22.371696 9048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "40e310befbf24f81ac694043f5e57232"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "81073ea5793b9fe3f05d2a0af0441bb0"
server_key_iv: "3daa55436cc4dc9d6b7c6d8d9da0b2b3"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.371806 9048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:22.385059 9048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:22.388121 9048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:22.388342 9048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:22.395831 9048 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43743
I20260810 22:57:22.395918 9123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43743 every 8 connection(s)
I20260810 22:57:22.396777 9048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:57:22.399518 9124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:57:22.404759 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232: Bootstrap starting.
I20260810 22:57:22.405184 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 9048
I20260810 22:57:22.405314 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:57:22.405651 30010 external_mini_cluster.cc:1428] Setting key ab2d148f5311b5c9da770020da6e319a
I20260810 22:57:22.407539 9124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232: Neither blocks nor log segments found. Creating new log.
I20260810 22:57:22.408267 9124 log.cc:824] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:22.410596 9124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232: No bootstrap required, opened a new log
Aug 10 22:57:22 dist-test-slave-6nmv krb5kdc[9034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:57:22.414892 9124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e310befbf24f81ac694043f5e57232" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43743 } }
I20260810 22:57:22.415112 9124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:57:22.415186 9124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e310befbf24f81ac694043f5e57232, State: Initialized, Role: FOLLOWER
I20260810 22:57:22.415671 9124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e310befbf24f81ac694043f5e57232" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43743 } }
I20260810 22:57:22.415850 9124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:57:22.415956 9124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:57:22.416079 9124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:57:22.416975 9124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e310befbf24f81ac694043f5e57232" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43743 } }
I20260810 22:57:22.417294 9124 leader_election.cc:302] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40e310befbf24f81ac694043f5e57232; no voters:
I20260810 22:57:22.417610 9124 leader_election.cc:288] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:57:22.417685 9129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:57:22.417903 9129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [term 1 LEADER]: Becoming Leader. State: Replica: 40e310befbf24f81ac694043f5e57232, State: Running, Role: LEADER
I20260810 22:57:22.418220 9129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e310befbf24f81ac694043f5e57232" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43743 } }
I20260810 22:57:22.418794 9124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:57:22.419874 9131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40e310befbf24f81ac694043f5e57232. Latest consensus state: current_term: 1 leader_uuid: "40e310befbf24f81ac694043f5e57232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e310befbf24f81ac694043f5e57232" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43743 } } }
I20260810 22:57:22.420252 9131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:22.420166 9130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40e310befbf24f81ac694043f5e57232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40e310befbf24f81ac694043f5e57232" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43743 } } }
I20260810 22:57:22.420579 9130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232 [sys.catalog]: This master's current role is: LEADER
I20260810 22:57:22.420660 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:22.406995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.29.78.190:43743)
0810 22:57:22.407394 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:22.407409 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:22.407442 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:57:22.408069 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:22.408086 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:22.408471 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:22.408836 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:22.409722 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.410745 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.411818 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.412303 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.412423 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:22.415138 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:22.415144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:22.415160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:22.415189 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:22.417146 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:22.417620 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:22.417625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:22.417630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:22.417760 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:22.418025 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:22.418029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:22.418031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:22.418461 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:22.418684 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:22.419313 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 22:57:22.419651 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":129,"threads_started":1}
I20260810 22:57:22.421111 9140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:57:22.424252 9140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:57:22.429813 9140 catalog_manager.cc:1347] Generated new cluster ID: 9628594c9a8343cd809cdfe9330087ed
I20260810 22:57:22.429911 9140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:57:22.443953 9140 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:57:22.444886 9140 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:57:22.448966 9140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232: Generated new TSK 0
I20260810 22:57:22.449697 9140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:57:22.476742 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43743
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:22.584121 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:22.584342 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:22.584378 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:22.588210 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:22.588259 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:22.588342 9153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:57:22.593313 9153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:22.594250 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:22.595244 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:22.604598 9160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:22.604664 9159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:22.604815 9162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:22.604943 9153 server_base.cc:1034] running on GCE node
I20260810 22:57:22.605502 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:22.606263 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:22.607446 9153 hybrid_clock.cc:648] HybridClock initialized: now 1786402642607421 us; error 37 us; skew 500 ppm
Aug 10 22:57:22 dist-test-slave-6nmv krb5kdc[9034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402642, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:22.610916 9153 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:57:22.611821 9153 webserver.cc:460] Webserver started at http://127.29.78.129:43979/ using document root <none> and password file <none>
I20260810 22:57:22.612363 9153 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:22.612426 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:22.612664 9153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:22.615012 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "69a94eb31e004742ab4346b70ecb4c78"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "1ec04bdbb506678c55e5fc4e66be499c"
server_key_iv: "8ec6c3311adef1c6c48a2481803ea406"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.615661 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "69a94eb31e004742ab4346b70ecb4c78"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "1ec04bdbb506678c55e5fc4e66be499c"
server_key_iv: "8ec6c3311adef1c6c48a2481803ea406"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.619791 9153 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260810 22:57:22.622493 9169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:22.623754 9153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:57:22.623943 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "69a94eb31e004742ab4346b70ecb4c78"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "1ec04bdbb506678c55e5fc4e66be499c"
server_key_iv: "8ec6c3311adef1c6c48a2481803ea406"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.624045 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:22.635474 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:22.638285 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:22.638499 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:22.639088 9153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:22.639940 9153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:22.639983 9153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:22.640017 9153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:22.640079 9153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:22.647979 9153 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:40787
I20260810 22:57:22.648041 9297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:40787 every 8 connection(s)
I20260810 22:57:22.648833 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:57:22.652464 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 9153
I20260810 22:57:22.652587 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:57:22.652881 30010 external_mini_cluster.cc:1428] Setting key 34ea61f19f2c4da67fcfd6644c9463b6
Aug 10 22:57:22 dist-test-slave-6nmv krb5kdc[9034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402642, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:57:22.663337 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:22.650624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:55893 (local address 127.29.78.190:43743)
0810 22:57:22.650746 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:22.650749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:22.651515 (+ 766us) server_negotiation.cc:407] Connection header received
0810 22:57:22.653029 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:22.653033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:22.653095 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:22.653197 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:22.654706 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.655258 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.656511 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.656791 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.656870 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:22.659724 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:22.659733 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:22.659735 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:22.659758 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:22.661189 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:22.661727 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:22.661731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:22.661733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:22.661881 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:22.662231 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:22.662236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:22.662238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:22.662425 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:22.662533 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:22.663062 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 22:57:22.663220 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 22:57:22.664179 9301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:22.650902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43743 (local address 127.29.78.129:55893)
0810 22:57:22.651369 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:22.651410 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:22.652804 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:22.653368 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:22.653375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:22.653814 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:22.654542 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:22.654557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.655392 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.655398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:22.656376 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:22.656389 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.656946 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.656953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:22.657152 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:22.657837 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:22.657870 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:22.659550 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:22.661325 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:22.661331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:22.661344 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:22.661601 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:22.661987 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:22.661990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:22.661991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:22.662135 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:22.662532 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:22.662539 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:22.662785 (+ 246us) client_negotiation.cc:769] Sending connection context
0810 22:57:22.662984 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 22:57:22.663253 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":139,"threads_started":1}
I20260810 22:57:22.665580 9298 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 22:57:22.665911 9298 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:22.666436 9298 heartbeater.cc:511] Master 127.29.78.190:43743 requested a full tablet report, sending...
I20260810 22:57:22.667997 9084 ts_manager.cc:194] Registered new tserver with Master: 69a94eb31e004742ab4346b70ecb4c78 (127.29.78.129:40787)
I20260810 22:57:22.669737 9084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:55893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:57:22.682379 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43743
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:22.787469 9306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:22.787680 9306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:22.787714 9306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:22.791374 9306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:22.791416 9306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:22.791512 9306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:57:22.796132 9306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:22.797032 9306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:22.798025 9306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:22.806984 9315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:22.807025 9312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:22.807090 9306 server_base.cc:1034] running on GCE node
W20260810 22:57:22.807034 9313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:22.807787 9306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:22.808380 9306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:22.809573 9306 hybrid_clock.cc:648] HybridClock initialized: now 1786402642809561 us; error 26 us; skew 500 ppm
Aug 10 22:57:22 dist-test-slave-6nmv krb5kdc[9034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402642, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:22.812533 9306 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:57:22.813498 9306 webserver.cc:460] Webserver started at http://127.29.78.130:41629/ using document root <none> and password file <none>
I20260810 22:57:22.814059 9306 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:22.814127 9306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:22.814286 9306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:22.816129 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "418f2bd025b346f6b0986c04bbb4bcf6"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "de64cbd10017f729c1746ba4def9a0b5"
server_key_iv: "64493eb8e3c11e53ee686e0b022bd723"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.816598 9306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "418f2bd025b346f6b0986c04bbb4bcf6"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "de64cbd10017f729c1746ba4def9a0b5"
server_key_iv: "64493eb8e3c11e53ee686e0b022bd723"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.820540 9306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:57:22.822815 9323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:22.823949 9306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:57:22.824070 9306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "418f2bd025b346f6b0986c04bbb4bcf6"
format_stamp: "Formatted at 2026-08-10 22:57:22 on dist-test-slave-6nmv"
server_key: "de64cbd10017f729c1746ba4def9a0b5"
server_key_iv: "64493eb8e3c11e53ee686e0b022bd723"
server_key_version: "encryptionkey@0"
I20260810 22:57:22.824174 9306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:22.848126 9306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:22.851400 9306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:22.851609 9306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:22.852264 9306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:22.853080 9306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:22.853124 9306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:22.853188 9306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:22.853224 9306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:22.861588 9306 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:35115
I20260810 22:57:22.861613 9446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:35115 every 8 connection(s)
I20260810 22:57:22.862522 9306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:57:22.868588 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 9306
I20260810 22:57:22.868705 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:57:22.869019 30010 external_mini_cluster.cc:1428] Setting key f44ee1fb2a3ddd03eb5e418ef4d38a9f
Aug 10 22:57:22 dist-test-slave-6nmv krb5kdc[9034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402642, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:57:22.876262 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:22.864160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:43091 (local address 127.29.78.190:43743)
0810 22:57:22.864291 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:22.864294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:22.865039 (+ 745us) server_negotiation.cc:407] Connection header received
0810 22:57:22.866357 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:22.866359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:22.866403 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:22.866473 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:22.867497 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.867972 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.869168 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.869447 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.869564 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:22.872333 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:22.872337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:22.872339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:22.872367 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:22.874117 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:22.874667 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:22.874670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:22.874671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:22.874770 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:22.875083 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:22.875088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:22.875089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:22.875300 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:22.875401 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:22.875974 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 22:57:22.876099 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:57:22.877030 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:22.864499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43743 (local address 127.29.78.130:43091)
0810 22:57:22.864895 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:22.864936 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:22.866177 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:22.866628 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:22.866634 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:22.866948 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:22.867364 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:22.867376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.868070 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.868072 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:22.869022 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:22.869032 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:22.869645 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:22.869649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:22.869936 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:22.870571 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:22.870593 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:22.872177 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:22.874235 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:22.874241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:22.874255 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:22.874532 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:22.874863 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:22.874866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:22.874868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:22.874995 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:22.875401 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:22.875408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:22.875687 (+ 279us) client_negotiation.cc:769] Sending connection context
0810 22:57:22.875905 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 22:57:22.876215 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260810 22:57:22.878212 9447 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43743
I20260810 22:57:22.878470 9447 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:22.878978 9447 heartbeater.cc:511] Master 127.29.78.190:43743 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:57:22.880050 9084 ts_manager.cc:194] Registered new tserver with Master: 418f2bd025b346f6b0986c04bbb4bcf6 (127.29.78.130:35115)
I20260810 22:57:22.880626 9084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:43091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:57:22.898388 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43743
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:57:23.004730 9454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:57:23.004943 9454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:57:23.004981 9454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:57:23.008708 9454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:57:23.008760 9454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:57:23.008814 9454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:57:23.013666 9454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:57:23.014567 9454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:57:23.015425 9454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:57:23.025256 9462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:23.025329 9464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:57:23.025336 9461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:57:23.025604 9454 server_base.cc:1034] running on GCE node
I20260810 22:57:23.026038 9454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:57:23.026631 9454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:57:23.027817 9454 hybrid_clock.cc:648] HybridClock initialized: now 1786402643027799 us; error 97 us; skew 500 ppm
Aug 10 22:57:23 dist-test-slave-6nmv krb5kdc[9034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402643, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:57:23.030751 9454 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:57:23.031603 9454 webserver.cc:460] Webserver started at http://127.29.78.131:37883/ using document root <none> and password file <none>
I20260810 22:57:23.032168 9454 fs_manager.cc:359] Metadata directory not provided
I20260810 22:57:23.032259 9454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:57:23.032500 9454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:57:23.034535 9454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "9bf08cd3abd647dcbad56a324162fc62"
format_stamp: "Formatted at 2026-08-10 22:57:23 on dist-test-slave-6nmv"
server_key: "b3b69d348f9ff1d12b4d1fe531bfd27c"
server_key_iv: "55f81037be89e34ebbd396c46b21fde9"
server_key_version: "encryptionkey@0"
I20260810 22:57:23.035140 9454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "9bf08cd3abd647dcbad56a324162fc62"
format_stamp: "Formatted at 2026-08-10 22:57:23 on dist-test-slave-6nmv"
server_key: "b3b69d348f9ff1d12b4d1fe531bfd27c"
server_key_iv: "55f81037be89e34ebbd396c46b21fde9"
server_key_version: "encryptionkey@0"
I20260810 22:57:23.038864 9454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:57:23.041246 9473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:23.042423 9454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:57:23.042562 9454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "9bf08cd3abd647dcbad56a324162fc62"
format_stamp: "Formatted at 2026-08-10 22:57:23 on dist-test-slave-6nmv"
server_key: "b3b69d348f9ff1d12b4d1fe531bfd27c"
server_key_iv: "55f81037be89e34ebbd396c46b21fde9"
server_key_version: "encryptionkey@0"
I20260810 22:57:23.042663 9454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:57:23.061545 9454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:57:23.064560 9454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:57:23.064738 9454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:57:23.065235 9454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:57:23.066030 9454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:57:23.066077 9454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:23.066129 9454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:57:23.066159 9454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:57:23.074432 9454 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:46449
I20260810 22:57:23.074491 9596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:46449 every 8 connection(s)
I20260810 22:57:23.075420 9454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
Aug 10 22:57:23 dist-test-slave-6nmv krb5kdc[9034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402643, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:57:23.084949 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 9454
I20260810 22:57:23.085079 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:57:23.085366 30010 external_mini_cluster.cc:1428] Setting key 999cb71ea5b5dbfb016735cf1b95f856
I20260810 22:57:23.089159 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.077298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:44517 (local address 127.29.78.190:43743)
0810 22:57:23.077467 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.077472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.078181 (+ 709us) server_negotiation.cc:407] Connection header received
0810 22:57:23.079532 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.079534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.079583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.079657 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:23.080763 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.081254 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.082396 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.082596 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.082677 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.085156 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:23.085162 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:23.085165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:23.085195 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:23.087071 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:23.087621 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:23.087626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:23.087628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:23.087729 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:23.088131 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:23.088134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:23.088135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:23.088335 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:23.088433 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.088923 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.089040 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 22:57:23.089936 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.077554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43743 (local address 127.29.78.131:44517)
0810 22:57:23.078045 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:23.078084 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:23.079321 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:23.079821 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:23.079827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:23.080164 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:57:23.080613 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.080626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.081379 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.081381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.082276 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.082285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.082730 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.082733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.082889 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.083474 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:57:23.083491 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:57:23.084921 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:57:23.087200 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:23.087204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:23.087214 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:23.087523 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:23.087817 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:57:23.087819 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:57:23.087820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:57:23.087985 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:57:23.088447 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:57:23.088452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:57:23.088663 (+ 211us) client_negotiation.cc:769] Sending connection context
0810 22:57:23.088813 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 22:57:23.089093 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":111,"threads_started":1}
I20260810 22:57:23.090991 9599 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43743
I20260810 22:57:23.091231 9599 heartbeater.cc:464] Registering TS with master...
I20260810 22:57:23.091699 9599 heartbeater.cc:511] Master 127.29.78.190:43743 requested a full tablet report, sending...
I20260810 22:57:23.092653 9084 ts_manager.cc:194] Registered new tserver with Master: 9bf08cd3abd647dcbad56a324162fc62 (127.29.78.131:46449)
I20260810 22:57:23.093145 9084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:44517
I20260810 22:57:23.099125 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:57:23.112109 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.100891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.29.78.190:43743)
0810 22:57:23.101026 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.101029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.101210 (+ 181us) server_negotiation.cc:407] Connection header received
0810 22:57:23.101294 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.101296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.101328 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.101398 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:57:23.102102 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.102557 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.103296 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.103480 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.103552 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.109264 (+ 5712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:57:23.109268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:57:23.109270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:57:23.109292 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:57:23.110664 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:23.111057 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:23.111060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:23.111061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:23.111134 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:57:23.111359 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:57:23.111361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:57:23.111362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:57:23.111556 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:57:23.111687 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.111911 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.112001 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:57:23.114763 9084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:57:23.117617 9084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:57:23.132114 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.126185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40787 (local address 127.0.0.1:37722)
0810 22:57:23.126445 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:23.126459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:23.126569 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:23.127557 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:23.127562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:23.127583 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:57:23.127684 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.127689 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.129367 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.129370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.130606 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.130615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.131516 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.131520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.131641 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.131686 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 22:57:23.131805 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 22:57:23.131935 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":174,"thread_start_us":131,"threads_started":1}
I20260810 22:57:23.132159 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.126667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.29.78.129:40787)
0810 22:57:23.127079 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.127093 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.127116 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:57:23.127195 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.127200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.127416 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.127557 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:57:23.127812 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.129194 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.130745 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.131381 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.131494 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.131639 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.131864 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.131972 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":120,"threads_started":1}
I20260810 22:57:23.132517 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.126368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46449 (local address 127.0.0.1:34306)
0810 22:57:23.126841 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:23.126853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:23.126948 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:23.128456 (+ 1508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:23.128459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:23.128476 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:57:23.128557 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.128563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.129654 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.129658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.131095 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.131103 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.132122 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.132125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.132219 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.132249 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 22:57:23.132303 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 22:57:23.132373 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":201,"thread_start_us":87,"threads_started":1}
I20260810 22:57:23.132737 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.125717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:35115 (local address 127.0.0.1:48644)
0810 22:57:23.126197 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:23.126242 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:23.126366 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:23.127782 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:23.127785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:23.127803 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:57:23.127913 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.127919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.129414 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.129420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.131086 (+ 1666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.131098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.132185 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.132192 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.132288 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.132325 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 22:57:23.132394 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 22:57:23.132471 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":199,"thread_start_us":60,"threads_started":1}
I20260810 22:57:23.133322 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.125881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.29.78.130:35115)
0810 22:57:23.127349 (+ 1468us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.127359 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.127381 (+ 22us) server_negotiation.cc:407] Connection header received
0810 22:57:23.127454 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.127461 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.127637 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.127802 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:57:23.128051 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.129277 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.131243 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.132032 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.132887 (+ 855us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.133028 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.133080 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.133166 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1376,"thread_start_us":120,"threads_started":1}
I20260810 22:57:23.133589 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.126803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34306 (local address 127.29.78.131:46449)
0810 22:57:23.128032 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.128045 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.128068 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:57:23.128142 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.128145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.128322 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.128476 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:57:23.128684 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.129498 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.131221 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.131983 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.133074 (+ 1091us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.133192 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.133234 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.133308 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260810 22:57:23.135176 9227 tablet_service.cc:1467] Processing CreateTablet for tablet 705703a5eb5041e2998bf528bc00a4b1 (DEFAULT_TABLE table=test-table [id=a7025b18304c4a2ca984d8ed71baf2f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:57:23.135644 9377 tablet_service.cc:1467] Processing CreateTablet for tablet 705703a5eb5041e2998bf528bc00a4b1 (DEFAULT_TABLE table=test-table [id=a7025b18304c4a2ca984d8ed71baf2f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:57:23.136323 9227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705703a5eb5041e2998bf528bc00a4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:57:23.136008 9528 tablet_service.cc:1467] Processing CreateTablet for tablet 705703a5eb5041e2998bf528bc00a4b1 (DEFAULT_TABLE table=test-table [id=a7025b18304c4a2ca984d8ed71baf2f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:57:23.136641 9377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705703a5eb5041e2998bf528bc00a4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:57:23.136896 9528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705703a5eb5041e2998bf528bc00a4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:57:23.142779 9622 tablet_bootstrap.cc:492] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Bootstrap starting.
I20260810 22:57:23.143905 9624 tablet_bootstrap.cc:492] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78: Bootstrap starting.
I20260810 22:57:23.145236 9622 tablet_bootstrap.cc:654] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Neither blocks nor log segments found. Creating new log.
I20260810 22:57:23.145452 9624 tablet_bootstrap.cc:654] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78: Neither blocks nor log segments found. Creating new log.
I20260810 22:57:23.146130 9622 log.cc:824] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:23.146145 9623 tablet_bootstrap.cc:492] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Bootstrap starting.
I20260810 22:57:23.146690 9624 log.cc:824] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:23.148082 9623 tablet_bootstrap.cc:654] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Neither blocks nor log segments found. Creating new log.
I20260810 22:57:23.148308 9622 tablet_bootstrap.cc:492] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: No bootstrap required, opened a new log
I20260810 22:57:23.148516 9622 ts_tablet_manager.cc:1397] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:57:23.148836 9624 tablet_bootstrap.cc:492] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78: No bootstrap required, opened a new log
I20260810 22:57:23.149032 9624 ts_tablet_manager.cc:1397] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 22:57:23.149223 9623 log.cc:824] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Log is configured to *not* fsync() on all Append() calls
I20260810 22:57:23.151149 9623 tablet_bootstrap.cc:492] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: No bootstrap required, opened a new log
I20260810 22:57:23.151309 9623 ts_tablet_manager.cc:1397] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:57:23.152062 9622 raft_consensus.cc:348] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.152299 9622 raft_consensus.cc:374] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:57:23.152370 9622 raft_consensus.cc:729] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418f2bd025b346f6b0986c04bbb4bcf6, State: Initialized, Role: FOLLOWER
I20260810 22:57:23.152827 9622 consensus_queue.cc:260] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.152747 9624 raft_consensus.cc:348] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.153002 9624 raft_consensus.cc:374] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:57:23.153060 9624 raft_consensus.cc:729] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a94eb31e004742ab4346b70ecb4c78, State: Initialized, Role: FOLLOWER
I20260810 22:57:23.153460 9624 consensus_queue.cc:260] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.153878 9622 ts_tablet_manager.cc:1428] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 22:57:23.154224 9447 heartbeater.cc:503] Master 127.29.78.190:43743 was elected leader, sending a full tablet report...
I20260810 22:57:23.154383 9624 ts_tablet_manager.cc:1428] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:57:23.154896 9623 raft_consensus.cc:348] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.155059 9623 raft_consensus.cc:374] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:57:23.155092 9623 raft_consensus.cc:729] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf08cd3abd647dcbad56a324162fc62, State: Initialized, Role: FOLLOWER
I20260810 22:57:23.155293 9298 heartbeater.cc:503] Master 127.29.78.190:43743 was elected leader, sending a full tablet report...
I20260810 22:57:23.155607 9623 consensus_queue.cc:260] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.156152 9599 heartbeater.cc:503] Master 127.29.78.190:43743 was elected leader, sending a full tablet report...
I20260810 22:57:23.156423 9623 ts_tablet_manager.cc:1428] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260810 22:57:23.239214 9629 raft_consensus.cc:482] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:57:23.239460 9629 raft_consensus.cc:504] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.240661 9629 leader_election.cc:288] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9bf08cd3abd647dcbad56a324162fc62 (127.29.78.131:46449), 69a94eb31e004742ab4346b70ecb4c78 (127.29.78.129:40787)
I20260810 22:57:23.244514 9621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.241111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:37367 (local address 127.29.78.131:46449)
0810 22:57:23.241254 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.241258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.241277 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:57:23.241491 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.241494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.241541 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.241631 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:57:23.241976 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.242465 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.243498 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.243943 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.244067 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.244121 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.244314 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.244394 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 22:57:23.244514 9449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.240993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46449 (local address 127.29.78.130:37367)
0810 22:57:23.241211 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:23.241230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:23.241333 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:23.241643 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:23.241646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:23.241664 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:57:23.241790 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.241796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.242583 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.242590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.243361 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.243370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.244067 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.244071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.244177 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.244206 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:57:23.244298 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 22:57:23.244372 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260810 22:57:23.244642 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.241174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:41749 (local address 127.29.78.129:40787)
0810 22:57:23.241287 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.241292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.241573 (+ 281us) server_negotiation.cc:407] Connection header received
0810 22:57:23.241714 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.241719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.241769 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.241891 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:57:23.242144 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.242595 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.243542 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.243909 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.244031 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.244082 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.244443 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.244509 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:57:23.244978 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.240993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40787 (local address 127.29.78.130:41749)
0810 22:57:23.241470 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:23.241480 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 22:57:23.241603 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:57:23.241901 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:57:23.241904 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:57:23.241919 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:57:23.242034 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.242039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.242699 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.242703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.243424 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:57:23.243432 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.244181 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.244185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:57:23.244284 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.244311 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:57:23.244777 (+ 466us) client_negotiation.cc:240] Negotiation successful
0810 22:57:23.244860 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":33,"thread_start_us":152,"threads_started":1}
I20260810 22:57:23.245292 9548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705703a5eb5041e2998bf528bc00a4b1" candidate_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf08cd3abd647dcbad56a324162fc62" is_pre_election: true
I20260810 22:57:23.245679 9548 raft_consensus.cc:2454] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418f2bd025b346f6b0986c04bbb4bcf6 in term 0.
I20260810 22:57:23.245710 9247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705703a5eb5041e2998bf528bc00a4b1" candidate_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a94eb31e004742ab4346b70ecb4c78" is_pre_election: true
I20260810 22:57:23.246070 9247 raft_consensus.cc:2454] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418f2bd025b346f6b0986c04bbb4bcf6 in term 0.
I20260810 22:57:23.246250 9327 leader_election.cc:302] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418f2bd025b346f6b0986c04bbb4bcf6, 9bf08cd3abd647dcbad56a324162fc62; no voters:
I20260810 22:57:23.246574 9629 raft_consensus.cc:2781] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:57:23.246685 9629 raft_consensus.cc:482] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:57:23.246752 9629 raft_consensus.cc:3037] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:57:23.247851 9629 raft_consensus.cc:504] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.248250 9629 leader_election.cc:288] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [CANDIDATE]: Term 1 election: Requested vote from peers 9bf08cd3abd647dcbad56a324162fc62 (127.29.78.131:46449), 69a94eb31e004742ab4346b70ecb4c78 (127.29.78.129:40787)
I20260810 22:57:23.248631 9548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705703a5eb5041e2998bf528bc00a4b1" candidate_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bf08cd3abd647dcbad56a324162fc62"
I20260810 22:57:23.248668 9247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705703a5eb5041e2998bf528bc00a4b1" candidate_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a94eb31e004742ab4346b70ecb4c78"
I20260810 22:57:23.248799 9247 raft_consensus.cc:3037] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:57:23.248795 9548 raft_consensus.cc:3037] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:57:23.249917 9247 raft_consensus.cc:2454] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418f2bd025b346f6b0986c04bbb4bcf6 in term 1.
I20260810 22:57:23.249986 9548 raft_consensus.cc:2454] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418f2bd025b346f6b0986c04bbb4bcf6 in term 1.
I20260810 22:57:23.250231 9325 leader_election.cc:302] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418f2bd025b346f6b0986c04bbb4bcf6, 69a94eb31e004742ab4346b70ecb4c78; no voters:
I20260810 22:57:23.250476 9629 raft_consensus.cc:2781] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:57:23.250742 9629 raft_consensus.cc:686] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 1 LEADER]: Becoming Leader. State: Replica: 418f2bd025b346f6b0986c04bbb4bcf6, State: Running, Role: LEADER
I20260810 22:57:23.251116 9629 consensus_queue.cc:237] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } }
I20260810 22:57:23.254424 9084 catalog_manager.cc:5495] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 418f2bd025b346f6b0986c04bbb4bcf6 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418f2bd025b346f6b0986c04bbb4bcf6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 35115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 } health_report { overall_health: UNKNOWN } } }
I20260810 22:57:23.282078 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.278849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48648 (local address 127.29.78.130:35115)
0810 22:57:23.278984 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:57:23.278988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:57:23.279001 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:57:23.279073 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:57:23.279075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:57:23.279129 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:57:23.279220 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:57:23.279490 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.280022 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.280856 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:57:23.281054 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:57:23.281130 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:57:23.281408 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:57:23.281774 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:57:23.281898 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 22:57:23.281919 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 22:57:23.281965 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:57:23.287581 9247 raft_consensus.cc:1261] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 1 FOLLOWER]: Refusing update from remote peer 418f2bd025b346f6b0986c04bbb4bcf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:57:23.287580 9548 raft_consensus.cc:1261] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 1 FOLLOWER]: Refusing update from remote peer 418f2bd025b346f6b0986c04bbb4bcf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:57:23.288309 9633 consensus_queue.cc:1035] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a94eb31e004742ab4346b70ecb4c78" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:57:23.288570 9629 consensus_queue.cc:1035] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bf08cd3abd647dcbad56a324162fc62" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:57:23.296236 9638 mvcc.cc:205] Tried to move back new op lower bound from 7317105226898681856 to 7317105226759389184. Current Snapshot: MvccSnapshot[applied={T|T < 7317105226898681856}]
I20260810 22:57:23.296289 9637 mvcc.cc:205] Tried to move back new op lower bound from 7317105226898681856 to 7317105226759389184. Current Snapshot: MvccSnapshot[applied={T|T < 7317105226898681856}]
I20260810 22:57:23.304534 9084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:57:23.304754 9084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:57:23.307323 9084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40e310befbf24f81ac694043f5e57232: Sending DeleteTablet for 3 replicas of tablet 705703a5eb5041e2998bf528bc00a4b1
I20260810 22:57:23.308151 9377 tablet_service.cc:1514] Processing DeleteTablet for tablet 705703a5eb5041e2998bf528bc00a4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:57:23 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:48644
I20260810 22:57:23.308151 9528 tablet_service.cc:1514] Processing DeleteTablet for tablet 705703a5eb5041e2998bf528bc00a4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:57:23 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:34306
I20260810 22:57:23.308200 9227 tablet_service.cc:1514] Processing DeleteTablet for tablet 705703a5eb5041e2998bf528bc00a4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:57:23 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:37722
I20260810 22:57:23.308728 9648 tablet_replica.cc:330] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78: stopping tablet replica
I20260810 22:57:23.308851 9646 tablet_replica.cc:330] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: stopping tablet replica
I20260810 22:57:23.309083 9648 raft_consensus.cc:2229] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:57:23.309083 9646 raft_consensus.cc:2229] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:57:23.309417 9648 raft_consensus.cc:2258] T 705703a5eb5041e2998bf528bc00a4b1 P 69a94eb31e004742ab4346b70ecb4c78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:57:23.309450 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 9153
I20260810 22:57:23.309633 9646 raft_consensus.cc:2258] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:57:23.311894 9646 ts_tablet_manager.cc:1905] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:57:23.314033 9646 ts_tablet_manager.cc:1918] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:57:23.314114 9646 log.cc:1196] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/705703a5eb5041e2998bf528bc00a4b1
I20260810 22:57:23.315691 9646 ts_tablet_manager.cc:1939] T 705703a5eb5041e2998bf528bc00a4b1 P 418f2bd025b346f6b0986c04bbb4bcf6: Deleting consensus metadata
I20260810 22:57:23.316949 9647 tablet_replica.cc:330] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: stopping tablet replica
I20260810 22:57:23.316985 9069 catalog_manager.cc:4841] TS 418f2bd025b346f6b0986c04bbb4bcf6 (127.29.78.130:35115): tablet 705703a5eb5041e2998bf528bc00a4b1 (table test-table [id=a7025b18304c4a2ca984d8ed71baf2f5]) successfully deleted
I20260810 22:57:23.317278 9647 raft_consensus.cc:2229] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:57:23.317586 9647 raft_consensus.cc:2258] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:57:23.319214 9647 ts_tablet_manager.cc:1905] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:57:23.322185 9067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:40787 (error 108)
I20260810 22:57:23.322419 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 9306
I20260810 22:57:23.324086 9647 ts_tablet_manager.cc:1918] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:57:23.324218 9647 log.cc:1196] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/705703a5eb5041e2998bf528bc00a4b1
I20260810 22:57:23.324548 9647 ts_tablet_manager.cc:1939] T 705703a5eb5041e2998bf528bc00a4b1 P 9bf08cd3abd647dcbad56a324162fc62: Deleting consensus metadata
I20260810 22:57:23.325729 9069 catalog_manager.cc:4841] TS 9bf08cd3abd647dcbad56a324162fc62 (127.29.78.131:46449): tablet 705703a5eb5041e2998bf528bc00a4b1 (table test-table [id=a7025b18304c4a2ca984d8ed71baf2f5]) successfully deleted
I20260810 22:57:23.326309 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:57:23.325954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40787 (local address 127.0.0.1:37734)
0810 22:57:23.326125 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 22:57:23.326209 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 22:57:23.326491 9067 catalog_manager.cc:4584] TS 69a94eb31e004742ab4346b70ecb4c78 (127.29.78.129:40787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 705703a5eb5041e2998bf528bc00a4b1: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40787: connect: Connection refused (error 111)
I20260810 22:57:23.330619 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 9454
I20260810 22:57:23.339401 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 9048
2026-08-10T22:57:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2283 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:57:23 dist-test-slave-6nmv krb5kdc[9652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:57:23 dist-test-slave-6nmv krb5kdc[9652](info): set up 2 sockets
Aug 10 22:57:23 dist-test-slave-6nmv krb5kdc[9652](info): commencing operation
krb5kdc: starting...
W20260810 22:57:24.361346 30010 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:57:24 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:57:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
I20260810 22:57:24.538585 30010 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskX35aNT/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/postgres -l logfile start
2026-08-10 22:57:26.915 UTC [9686] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 22:57:26.915 UTC [9686] LOG: listening on IPv4 address "127.29.78.148", port 51433
2026-08-10 22:57:26.918 UTC [9686] LOG: listening on Unix socket "/tmp/.s.PGSQL.51433"
2026-08-10 22:57:26.940 UTC [9689] LOG: database system was shut down at 2026-08-10 22:57:25 UTC
2026-08-10 22:57:26.943 UTC [9686] LOG: database system is ready to accept connections
I20260810 22:57:27.883124 30010 mini_postgres.cc:96] Postgres bound to 51433
2026-08-10 22:57:27.890 UTC [9699] FATAL: database "slave" does not exist
127.29.78.148:51433 - accepting connections
I20260810 22:57:27.891561 30010 mini_ranger.cc:161] Starting Ranger...
I20260810 22:57:27.906489 30010 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 22:57:28.370693 30010 mini_ranger.cc:87] Created ranger Postgres database
I20260810 22:57:28.370797 30010 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-admin
2026-08-10 22:57:28,689 [I] DB FLAVOR :POSTGRES
2026-08-10 22:57:28,690 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:57:28,690 [I] Checking connection..
2026-08-10 22:57:28,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 22:57:28,940 [I] Checking connection passed.
2026-08-10 22:57:28,940 [I] --------- Verifying version history table ---------
2026-08-10 22:57:28,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:57:29,187 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:57:29,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:57:29,434 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:57:29,435 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 22:57:29,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 22:57:29.668 UTC [9771] WARNING: there is no transaction in progress
2026-08-10 22:57:29,676 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 22:57:29,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:57:29,926 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 22:57:29,927 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:57:29,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 22:57:30,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 22:57:30,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6nmv.c.gcp-upstream.internal','N') ;"
2026-08-10 22:57:30,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:57:30,882 [I] Table x_portal_user does not exist in database ranger
2026-08-10 22:57:30,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:57:31,134 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 22:57:31,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:57:31,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:57:31,605 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 22:57:31,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 22:57:32.517 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.532 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.547 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.562 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.576 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.591 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.652 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.658 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.672 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.682 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.692 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.704 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.714 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.720 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.727 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:32.733 UTC [9895] WARNING: there is no transaction in progress
2026-08-10 22:57:33,339 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 22:57:33,339 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 22:57:33,571 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 22:57:33,572 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:57:33,819 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 22:57:33,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:57:34,073 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 22:57:34,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:57:34,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:57:34,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6nmv.c.gcp-upstream.internal';"
2026-08-10 22:57:34,785 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 22:57:34,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 22:57:35,033 [I] DB_PATCHES have already been applied
I20260810 22:57:35.038697 30010 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-admin:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskX35aNT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 22:57:35.038787 30010 mini_ranger.cc:193] Using host: 127.29.78.148
I20260810 22:57:35.042936 30010 mini_ranger.cc:227] Ranger admin URL: http://127.29.78.148:55319
Aug 10, 2026 10:57:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:57:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:57:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 10:57:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 10:57:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 10:57:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 10:57:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 22:57:36.170713 30010 mini_ranger.cc:160] Time spent starting Ranger: real 8.279s user 0.001s sys 0.011s
Aug 10, 2026 10:57:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 10:57:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 10:57:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 10:57:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 10:57:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 22:57:57.772425 30010 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.78.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 22:57:57 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402677, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 22:57:57.859913 30010 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 22:57:57.869731 30010 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 22:57:58.704674 30010 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 22:57:58.704775 30010 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms
I20260810 22:57:58.704918 30010 mini_ranger_kms.cc:227] Using postgres at 127.29.78.148:51433
2026-08-10 22:57:58,936 [I] DB FLAVOR :POSTGRES
2026-08-10 22:57:58,936 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:57:58,936 [I] Checking connection
2026-08-10 22:57:58,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 22:57:59,212 [I] connection success
2026-08-10 22:57:59,212 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 22:57:59,212 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 22:57:59,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 22:57:59,453 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 22:57:59,453 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:57:59,453 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 22:57:59,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:51433/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 22:57:59,691 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 22:58:00.147544 30010 mini_ranger_kms.cc:310] Created kms service
I20260810 22:58:00.344091 30010 mini_ranger_kms.cc:326] Created kudu user
I20260810 22:58:00.389518 30010 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 22:58:00 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402677, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:00.804843 30010 mini_ranger_kms.cc:384] Added ranger policy
I20260810 22:58:00.805006 30010 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms:/tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 22:58:00.805042 30010 mini_ranger_kms.cc:240] Using host: 127.29.78.148
I20260810 22:58:00.809170 30010 mini_ranger_kms.cc:276] Ranger KMS PID: 10093
I20260810 22:58:00.809306 30010 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.78.148:49299
Aug 10, 2026 10:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 10:58:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 10:58:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 10:58:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260810 22:58:03.119515 30010 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.260s user 0.007s sys 0.210s
I20260810 22:58:03.119652 30010 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 22:58:03.119761 30010 mini_ranger_kms.cc:401] 127.29.78.148:49299/kms/v1/keys
Aug 10 22:58:03 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786402683, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 22:58:03 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402683, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 10 22:58:03 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402683, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 10 22:58:06 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:58:06.169293 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34409
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34409
--ranger_config_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:49299/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:06.279222 10149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:06.279510 10149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:06.279582 10149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:06.283461 10149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:58:06.283540 10149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:06.283579 10149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:06.283634 10149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:58:06.283669 10149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:58:06.288933 10149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.78.148:49299/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34409
--ranger_config_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:06.290010 10149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:06.291006 10149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:06.299537 10158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:06.299587 10160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:06.299533 10157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:06.300091 10149 server_base.cc:1034] running on GCE node
I20260810 22:58:06.300680 10149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:06.301735 10149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:06.302928 10149 hybrid_clock.cc:648] HybridClock initialized: now 1786402686302912 us; error 40 us; skew 500 ppm
Aug 10 22:58:06 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402686, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:06.310122 10149 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:58:06.311057 10149 webserver.cc:460] Webserver started at http://127.29.78.190:33703/ using document root <none> and password file <none>
I20260810 22:58:06.311609 10149 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:06.311690 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:06.311910 10149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:06 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402686, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:06.362568 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "0d280dbc8adc42fd928aa7ebec11908f"
format_stamp: "Formatted at 2026-08-10 22:58:06 on dist-test-slave-6nmv"
server_key: "b3dcd4dbb9b70fdf84b37a549f180c8d"
server_key_iv: "59ad2cd4d489fb8938c4c7dba26ce83e"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:06.363232 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "0d280dbc8adc42fd928aa7ebec11908f"
format_stamp: "Formatted at 2026-08-10 22:58:06 on dist-test-slave-6nmv"
server_key: "b3dcd4dbb9b70fdf84b37a549f180c8d"
server_key_iv: "59ad2cd4d489fb8938c4c7dba26ce83e"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:06.367278 10149 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 22:58:06.380810 10167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:06.382256 10149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:58:06.382431 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "0d280dbc8adc42fd928aa7ebec11908f"
format_stamp: "Formatted at 2026-08-10 22:58:06 on dist-test-slave-6nmv"
server_key: "b3dcd4dbb9b70fdf84b37a549f180c8d"
server_key_iv: "59ad2cd4d489fb8938c4c7dba26ce83e"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:06.382570 10149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:06.395820 10149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:06.398877 10149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:06.399087 10149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:06.407452 10149 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:34409
I20260810 22:58:06.407514 10223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:34409 every 8 connection(s)
I20260810 22:58:06.408458 10149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:58:06.411324 10224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:06.416904 10224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f: Bootstrap starting.
I20260810 22:58:06.417508 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 10149
I20260810 22:58:06.417629 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:58:06.417968 30010 external_mini_cluster.cc:1428] Setting key 99f6fef1939d25f5ae99507eb53226a7
I20260810 22:58:06.419842 10224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:06.420693 10224 log.cc:824] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:06.422858 10224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f: No bootstrap required, opened a new log
Aug 10 22:58:06 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:06.427259 10224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d280dbc8adc42fd928aa7ebec11908f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34409 } }
I20260810 22:58:06.427508 10224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:06.427604 10224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d280dbc8adc42fd928aa7ebec11908f, State: Initialized, Role: FOLLOWER
I20260810 22:58:06.428056 10224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d280dbc8adc42fd928aa7ebec11908f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34409 } }
I20260810 22:58:06.428205 10224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:58:06.428304 10224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:58:06.428402 10224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:06.429417 10224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d280dbc8adc42fd928aa7ebec11908f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34409 } }
I20260810 22:58:06.429749 10224 leader_election.cc:302] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d280dbc8adc42fd928aa7ebec11908f; no voters:
I20260810 22:58:06.430162 10224 leader_election.cc:288] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:58:06.430270 10231 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:58:06.430557 10231 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [term 1 LEADER]: Becoming Leader. State: Replica: 0d280dbc8adc42fd928aa7ebec11908f, State: Running, Role: LEADER
I20260810 22:58:06.430907 10231 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d280dbc8adc42fd928aa7ebec11908f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34409 } }
I20260810 22:58:06.431479 10224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:58:06.432260 10231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d280dbc8adc42fd928aa7ebec11908f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d280dbc8adc42fd928aa7ebec11908f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34409 } } }
I20260810 22:58:06.432480 10231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [sys.catalog]: This master's current role is: LEADER
I20260810 22:58:06.432741 10231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d280dbc8adc42fd928aa7ebec11908f. Latest consensus state: current_term: 1 leader_uuid: "0d280dbc8adc42fd928aa7ebec11908f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d280dbc8adc42fd928aa7ebec11908f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34409 } } }
I20260810 22:58:06.432905 10231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f [sys.catalog]: This master's current role is: LEADER
I20260810 22:58:06.433063 10227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:06.419238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.29.78.190:34409)
0810 22:58:06.419688 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:06.419712 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:06.419774 (+ 62us) server_negotiation.cc:407] Connection header received
0810 22:58:06.420597 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:06.420618 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:06.421016 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:06.421368 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:06.422246 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:06.423361 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:06.424247 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:06.424718 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:06.424928 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:06.427466 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:06.427471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:06.427503 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:06.427539 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:06.429318 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:06.429868 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:06.429875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:06.429880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:06.430041 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:06.430461 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:06.430467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:06.430470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:06.431022 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:06.431211 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:06.431710 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 22:58:06.432023 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":136,"threads_started":1}
I20260810 22:58:06.433233 10241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:58:06.437536 10241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:58:06.438177 10224 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:58:06.450870 10241 catalog_manager.cc:1347] Generated new cluster ID: 6d24695f33cf4391b3ca331aa45b3f38
I20260810 22:58:06.450955 10241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:58:06.475162 10241 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:58:06.477424 10241 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:58:06.501796 10241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f: Generated new TSK 0
I20260810 22:58:06.502751 10241 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:58:07.187143 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:49299/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34409
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:07.350723 10274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:07.351022 10274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:07.351105 10274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:07.355134 10274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:07.355222 10274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:07.355325 10274 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:58:07.361377 10274 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.78.148:49299/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:07.362759 10274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:07.364486 10274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:07.375445 10283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:07.375626 10285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:07.375811 10274 server_base.cc:1034] running on GCE node
W20260810 22:58:07.375756 10282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:07.376283 10274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:07.376943 10274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:07.378120 10274 hybrid_clock.cc:648] HybridClock initialized: now 1786402687378095 us; error 43 us; skew 500 ppm
Aug 10 22:58:07 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:07.380929 10274 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:58:07.381891 10274 webserver.cc:460] Webserver started at http://127.29.78.129:36197/ using document root <none> and password file <none>
I20260810 22:58:07.382445 10274 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:07.382529 10274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:07.382750 10274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:07 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:07.396773 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "df117490ee2743b8a4b7922090969281"
format_stamp: "Formatted at 2026-08-10 22:58:07 on dist-test-slave-6nmv"
server_key: "8a07cbe7d6172bed18efd5aaeb0a8004"
server_key_iv: "7ddf0d57ce78df0fe542e77d110a4f0a"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:07.397517 10274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "df117490ee2743b8a4b7922090969281"
format_stamp: "Formatted at 2026-08-10 22:58:07 on dist-test-slave-6nmv"
server_key: "8a07cbe7d6172bed18efd5aaeb0a8004"
server_key_iv: "7ddf0d57ce78df0fe542e77d110a4f0a"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:07.402580 10274 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 22:58:07.427381 10295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:07.429085 10274 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:58:07.429317 10274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "df117490ee2743b8a4b7922090969281"
format_stamp: "Formatted at 2026-08-10 22:58:07 on dist-test-slave-6nmv"
server_key: "8a07cbe7d6172bed18efd5aaeb0a8004"
server_key_iv: "7ddf0d57ce78df0fe542e77d110a4f0a"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:07.429531 10274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:07.450762 10274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:07.454416 10274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:07.454677 10274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:07.455405 10274 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:07.456496 10274 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:07.456593 10274 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:07.456728 10274 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:07.456811 10274 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:07.471001 10274 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:43377
I20260810 22:58:07.472335 10274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:58:07.474586 10426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:43377 every 8 connection(s)
I20260810 22:58:07.479030 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 10274
I20260810 22:58:07.479167 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:58:07.479501 30010 external_mini_cluster.cc:1428] Setting key a02de1cdfc3d01c732c5ff80c120aa2e
Aug 10 22:58:07 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:07.490410 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:07.476448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54517 (local address 127.29.78.190:34409)
0810 22:58:07.478277 (+ 1829us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:07.478290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:07.478306 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:58:07.478352 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:07.478355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:07.478401 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:07.478524 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:07.479926 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.480765 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.482434 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.482771 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.482906 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:07.485954 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:07.485958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:07.485961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:07.485986 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:07.487856 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:07.488536 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:07.488540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:07.488541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:07.488640 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:07.489098 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:07.489100 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:07.489101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:07.489277 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:07.489354 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:07.490119 (+ 765us) server_negotiation.cc:299] Negotiation successful
0810 22:58:07.490265 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1752,"thread_start_us":113,"threads_started":1}
I20260810 22:58:07.491385 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:07.475312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34409 (local address 127.29.78.129:54517)
0810 22:58:07.476878 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:07.476931 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:07.477930 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:07.478750 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:07.478759 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:07.479104 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:07.479739 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:07.479759 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.480919 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.480923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:07.482277 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:07.482289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.483088 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.483092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:07.483290 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:07.484010 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:07.484031 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:07.485741 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:07.487993 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:07.488002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:07.488019 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:07.488397 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:07.488757 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:07.488762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:07.488767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:07.488978 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:07.489388 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:07.489398 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:07.489775 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 22:58:07.490030 (+ 255us) client_negotiation.cc:240] Negotiation successful
0810 22:58:07.490362 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1355,"thread_start_us":138,"threads_started":1}
I20260810 22:58:07.492811 10427 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34409
I20260810 22:58:07.493064 10427 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:07.493620 10427 heartbeater.cc:511] Master 127.29.78.190:34409 requested a full tablet report, sending...
I20260810 22:58:07.498699 10184 ts_manager.cc:194] Registered new tserver with Master: df117490ee2743b8a4b7922090969281 (127.29.78.129:43377)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:58:07.500123 10184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:54517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:58:07.520169 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:49299/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34409
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:58:07 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 22:58:07.643374 10437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:07.643635 10437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:07.643738 10437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:07.648051 10437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:07.648123 10437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:07.648222 10437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:58:07.653586 10437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.78.148:49299/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:07.654769 10437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:07.655786 10437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:07.664822 10445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:07.665073 10447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:07.664872 10444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:07.665948 10437 server_base.cc:1034] running on GCE node
I20260810 22:58:07.666556 10437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:07.667263 10437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:07.668565 10437 hybrid_clock.cc:648] HybridClock initialized: now 1786402687668535 us; error 38 us; skew 500 ppm
Aug 10 22:58:07 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:07.671907 10437 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:58:07.672919 10437 webserver.cc:460] Webserver started at http://127.29.78.130:46227/ using document root <none> and password file <none>
I20260810 22:58:07.673594 10437 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:07.673727 10437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:07.674072 10437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:07 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:07.694351 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "27cb5473eeab4ac2b5f476491539141a"
format_stamp: "Formatted at 2026-08-10 22:58:07 on dist-test-slave-6nmv"
server_key: "492ba72702b0474381a776d11d4431b1"
server_key_iv: "6155a9e7a38b78696cef86659a371a8c"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:07.697750 10437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "27cb5473eeab4ac2b5f476491539141a"
format_stamp: "Formatted at 2026-08-10 22:58:07 on dist-test-slave-6nmv"
server_key: "492ba72702b0474381a776d11d4431b1"
server_key_iv: "6155a9e7a38b78696cef86659a371a8c"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:07.703603 10437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 22:58:07.730147 10455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:07.731832 10437 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260810 22:58:07.731976 10437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "27cb5473eeab4ac2b5f476491539141a"
format_stamp: "Formatted at 2026-08-10 22:58:07 on dist-test-slave-6nmv"
server_key: "492ba72702b0474381a776d11d4431b1"
server_key_iv: "6155a9e7a38b78696cef86659a371a8c"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:07.732113 10437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:07.783249 10437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:07.787211 10437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:07.787457 10437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:07.788230 10437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:07.789312 10437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:07.789381 10437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:07.789441 10437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:07.789481 10437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:07.800252 10437 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:46011
I20260810 22:58:07.801501 10437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:58:07.805450 10581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:46011 every 8 connection(s)
I20260810 22:58:07.809122 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 10437
I20260810 22:58:07.809259 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:58:07.809608 30010 external_mini_cluster.cc:1428] Setting key 63018d0d289a6d69ab8d5cfb376e1b9b
Aug 10 22:58:07 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:07.822247 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:07.805649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44907 (local address 127.29.78.190:34409)
0810 22:58:07.805827 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:07.805832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:07.806228 (+ 396us) server_negotiation.cc:407] Connection header received
0810 22:58:07.807606 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:07.807610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:07.807670 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:07.807729 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:07.809524 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.810477 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.812574 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.812969 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.813061 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:07.816430 (+ 3369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:07.816433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:07.816436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:07.816463 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:07.819441 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:07.820111 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:07.820115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:07.820117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:07.820212 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:07.820688 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:07.820692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:07.820694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:07.820946 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:07.821047 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:07.821930 (+ 883us) server_negotiation.cc:299] Negotiation successful
0810 22:58:07.822067 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:58:07.824031 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:07.804454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34409 (local address 127.29.78.130:44907)
0810 22:58:07.806047 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:07.806090 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:07.807328 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:07.808173 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:07.808182 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:07.808534 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:07.809256 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:07.809279 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.810867 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.810871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:07.812408 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:07.812421 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:07.813276 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:07.813280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:07.813496 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:07.814318 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:07.814341 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:07.816209 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:07.819586 (+ 3377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:07.819592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:07.819618 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:07.819981 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:07.820320 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:07.820323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:07.820325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:07.820537 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:07.821104 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:07.821113 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:07.821462 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 22:58:07.821686 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 22:58:07.822939 (+ 1253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"thread_start_us":136,"threads_started":1}
I20260810 22:58:07.825662 10582 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34409
I20260810 22:58:07.825948 10582 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:07.826553 10582 heartbeater.cc:511] Master 127.29.78.190:34409 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260810 22:58:07.827670 10184 ts_manager.cc:194] Registered new tserver with Master: 27cb5473eeab4ac2b5f476491539141a (127.29.78.130:46011)
I20260810 22:58:07.828418 10184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:44907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:58:07.845131 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:49299/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34409
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:07.983496 10590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:07.983791 10590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:07.983933 10590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:07.988662 10590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:07.988772 10590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:07.988889 10590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:58:07.996201 10590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.78.148:49299/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:07.997591 10590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:07.998711 10590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:08.010421 10599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:08.010416 10596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:08.012162 10590 server_base.cc:1034] running on GCE node
W20260810 22:58:08.012414 10597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:08.012953 10590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:08.013675 10590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:08.014842 10590 hybrid_clock.cc:648] HybridClock initialized: now 1786402688014757 us; error 99 us; skew 500 ppm
Aug 10 22:58:08 dist-test-slave-6nmv krb5kdc[9652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402688, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:08.018508 10590 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:58:08.019469 10590 webserver.cc:460] Webserver started at http://127.29.78.131:35067/ using document root <none> and password file <none>
I20260810 22:58:08.020083 10590 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:08.020143 10590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:08.020335 10590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:08 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402688, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:08.036170 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "caaad23d2a704b4192a8e3bed68510bc"
format_stamp: "Formatted at 2026-08-10 22:58:08 on dist-test-slave-6nmv"
server_key: "3a999e83684cfae3a988b32d22db57b8"
server_key_iv: "34f709722ffef81f6706a48ed97e4e8d"
server_key_version: "kuduclusterkey@0"
Aug 10 22:58:08 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:08.040537 10590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "caaad23d2a704b4192a8e3bed68510bc"
format_stamp: "Formatted at 2026-08-10 22:58:08 on dist-test-slave-6nmv"
server_key: "3a999e83684cfae3a988b32d22db57b8"
server_key_iv: "34f709722ffef81f6706a48ed97e4e8d"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:08.054297 10590 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.001s sys 0.004s
I20260810 22:58:08.071267 10607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:08.073007 10590 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 22:58:08.073228 10590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "caaad23d2a704b4192a8e3bed68510bc"
format_stamp: "Formatted at 2026-08-10 22:58:08 on dist-test-slave-6nmv"
server_key: "3a999e83684cfae3a988b32d22db57b8"
server_key_iv: "34f709722ffef81f6706a48ed97e4e8d"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:08.073536 10590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:08.089916 10590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:08.093164 10590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:08.093387 10590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:08.094146 10590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:08.095163 10590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:08.095249 10590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:08.095348 10590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:08.095404 10590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:08.114333 10590 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:46185
I20260810 22:58:08.115623 10590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:58:08.116514 10736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:46185 every 8 connection(s)
I20260810 22:58:08.124948 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 10590
I20260810 22:58:08.125090 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:58:08.125417 30010 external_mini_cluster.cc:1428] Setting key 10b3b4a94266d0c983a2990708f17d92
Aug 10 22:58:08 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402688, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:08.131711 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.117719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:59829 (local address 127.29.78.190:34409)
0810 22:58:08.117875 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.117880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.118936 (+ 1056us) server_negotiation.cc:407] Connection header received
0810 22:58:08.120059 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.120062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.120128 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.120233 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:08.121534 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.122409 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.124042 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.124407 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.124531 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.127331 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:08.127335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:08.127338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:08.127380 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:08.129219 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:08.129923 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:08.129927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:08.129929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:08.130016 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:08.130441 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:08.130444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:08.130445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:08.130654 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:08.130745 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.131388 (+ 643us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.131514 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:58:08.132750 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.118245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:34409 (local address 127.29.78.131:59829)
0810 22:58:08.118786 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.118838 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:08.119815 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:08.120418 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:08.120426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:08.120751 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:08.121353 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.121380 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.122540 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.122544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.123899 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.123909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.124563 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.124566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.124748 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.125490 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:08.125508 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:08.127135 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:08.129358 (+ 2223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:08.129364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:08.129379 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:08.129758 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:08.130125 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:08.130128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:08.130130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:08.130324 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:08.130769 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:08.130775 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:08.131097 (+ 322us) client_negotiation.cc:769] Sending connection context
0810 22:58:08.131316 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 22:58:08.131683 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":156,"threads_started":1}
I20260810 22:58:08.133914 10737 heartbeater.cc:347] Connected to a master server at 127.29.78.190:34409
I20260810 22:58:08.134169 10737 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:08.134692 10737 heartbeater.cc:511] Master 127.29.78.190:34409 requested a full tablet report, sending...
I20260810 22:58:08.135918 10184 ts_manager.cc:194] Registered new tserver with Master: caaad23d2a704b4192a8e3bed68510bc (127.29.78.131:46185)
I20260810 22:58:08.136719 10184 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:59829
I20260810 22:58:08.139530 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:58:08.154723 10431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.142017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.29.78.190:34409)
0810 22:58:08.142138 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.142142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.146199 (+ 4057us) server_negotiation.cc:407] Connection header received
0810 22:58:08.146336 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.146339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.146398 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.146486 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:08.147531 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.148061 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.149004 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.149309 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.149445 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.151009 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:08.151014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:08.151017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:08.151052 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:08.152648 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:08.153149 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:08.153156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:08.153158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:08.153253 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:08.153617 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:08.153622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:08.153623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:08.153912 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:08.154012 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.154385 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.154515 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 22:58:08.158124 10184 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:58:08.161351 10184 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:58:08.182307 10751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.171419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46011 (local address 127.0.0.1:55426)
0810 22:58:08.174361 (+ 2942us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.174401 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:08.174567 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:08.175398 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:08.175404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:08.175428 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:08.175568 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.175576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.177165 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.177168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.178670 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.178680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.181577 (+ 2897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.181582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.181768 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.181862 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 22:58:08.181977 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 22:58:08.182108 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2741,"mutex_wait_us":52,"thread_start_us":112,"threads_started":1}
I20260810 22:58:08.182909 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.172395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.29.78.131:46185)
0810 22:58:08.173155 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.173170 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.175384 (+ 2214us) server_negotiation.cc:407] Connection header received
0810 22:58:08.175468 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.175474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.175678 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.176533 (+ 855us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:08.177003 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.178186 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.180931 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.181719 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.181867 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.182020 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.182608 (+ 588us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.182723 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":186,"threads_started":1}
I20260810 22:58:08.182514 10750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.172177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:46185 (local address 127.0.0.1:57518)
0810 22:58:08.175176 (+ 2999us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.175190 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:08.175275 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:08.176709 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:08.176712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:08.176730 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:08.176856 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.176863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.178335 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.178338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.180721 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.180733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.182084 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.182088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.182193 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.182217 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:58:08.182292 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 22:58:08.182367 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2942,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260810 22:58:08.183512 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.172585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.29.78.130:46011)
0810 22:58:08.174083 (+ 1498us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.174093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.174504 (+ 411us) server_negotiation.cc:407] Connection header received
0810 22:58:08.174748 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.174756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.175006 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.175174 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:08.175889 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.177029 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.178861 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.181219 (+ 2358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.181333 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.181504 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.183216 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.183327 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1401,"thread_start_us":96,"threads_started":1}
I20260810 22:58:08.186821 10501 tablet_service.cc:1467] Processing CreateTablet for tablet 73082685fc694096af03af2ac083fda7 (DEFAULT_TABLE table=test-table [id=00f77f4b988a4264850e6087693ca23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:58:08.186821 10643 tablet_service.cc:1467] Processing CreateTablet for tablet 73082685fc694096af03af2ac083fda7 (DEFAULT_TABLE table=test-table [id=00f77f4b988a4264850e6087693ca23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:58:08.188015 10643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73082685fc694096af03af2ac083fda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:08.188015 10501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73082685fc694096af03af2ac083fda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:08.191334 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.171538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43377 (local address 127.0.0.1:39156)
0810 22:58:08.174876 (+ 3338us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.174891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:08.174980 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:08.186641 (+ 11661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:08.186645 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:08.186670 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:08.186797 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.186805 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.188541 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.188547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.189844 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.189856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.190929 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.190934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.191055 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.191081 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:58:08.191134 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 22:58:08.191198 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3240,"thread_start_us":263,"threads_started":1}
I20260810 22:58:08.192090 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.172275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.29.78.129:43377)
0810 22:58:08.186052 (+ 13777us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.186058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.186086 (+ 28us) server_negotiation.cc:407] Connection header received
0810 22:58:08.186206 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.186211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.186437 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.186978 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:08.187148 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.188357 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.190016 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.190787 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.191739 (+ 952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.191870 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.191910 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.191979 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13644,"thread_start_us":93,"threads_started":1}
I20260810 22:58:08.195079 10352 tablet_service.cc:1467] Processing CreateTablet for tablet 73082685fc694096af03af2ac083fda7 (DEFAULT_TABLE table=test-table [id=00f77f4b988a4264850e6087693ca23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:58:08.196238 10352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73082685fc694096af03af2ac083fda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:08.206331 10757 tablet_bootstrap.cc:492] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: Bootstrap starting.
I20260810 22:58:08.206902 10758 tablet_bootstrap.cc:492] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281: Bootstrap starting.
I20260810 22:58:08.209264 10757 tablet_bootstrap.cc:654] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:08.209272 10758 tablet_bootstrap.cc:654] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:08.210129 10759 tablet_bootstrap.cc:492] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Bootstrap starting.
I20260810 22:58:08.210167 10757 log.cc:824] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:08.210780 10758 log.cc:824] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:08.212453 10757 tablet_bootstrap.cc:492] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: No bootstrap required, opened a new log
I20260810 22:58:08.212631 10757 ts_tablet_manager.cc:1397] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:58:08.213322 10758 tablet_bootstrap.cc:492] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281: No bootstrap required, opened a new log
I20260810 22:58:08.213500 10758 ts_tablet_manager.cc:1397] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 22:58:08.216421 10757 raft_consensus.cc:348] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.216660 10757 raft_consensus.cc:374] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:08.216701 10757 raft_consensus.cc:729] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cb5473eeab4ac2b5f476491539141a, State: Initialized, Role: FOLLOWER
I20260810 22:58:08.217160 10757 consensus_queue.cc:260] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.218147 10757 ts_tablet_manager.cc:1428] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 22:58:08.218112 10758 raft_consensus.cc:348] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.218371 10758 raft_consensus.cc:374] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:08.218428 10758 raft_consensus.cc:729] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df117490ee2743b8a4b7922090969281, State: Initialized, Role: FOLLOWER
I20260810 22:58:08.218501 10582 heartbeater.cc:503] Master 127.29.78.190:34409 was elected leader, sending a full tablet report...
I20260810 22:58:08.219906 10758 consensus_queue.cc:260] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.221146 10758 ts_tablet_manager.cc:1428] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 22:58:08.221534 10427 heartbeater.cc:503] Master 127.29.78.190:34409 was elected leader, sending a full tablet report...
I20260810 22:58:08.225644 10759 tablet_bootstrap.cc:654] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:08.226630 10759 log.cc:824] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Log is configured to *not* fsync() on all Append() calls
W20260810 22:58:08.227205 10429 tablet.cc:2369] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:58:08.229058 10759 tablet_bootstrap.cc:492] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: No bootstrap required, opened a new log
I20260810 22:58:08.229269 10759 ts_tablet_manager.cc:1397] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20260810 22:58:08.233777 10759 raft_consensus.cc:348] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.234175 10759 raft_consensus.cc:374] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:08.234254 10759 raft_consensus.cc:729] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caaad23d2a704b4192a8e3bed68510bc, State: Initialized, Role: FOLLOWER
I20260810 22:58:08.234766 10759 consensus_queue.cc:260] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.235972 10759 ts_tablet_manager.cc:1428] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 22:58:08.236366 10737 heartbeater.cc:503] Master 127.29.78.190:34409 was elected leader, sending a full tablet report...
W20260810 22:58:08.305258 10585 tablet.cc:2369] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:58:08.367339 10738 tablet.cc:2369] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 22:58:08 dist-test-slave-6nmv krb5kdc[9652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402687, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:08.429659 10765 raft_consensus.cc:482] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:58:08.430006 10765 raft_consensus.cc:504] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.431555 10765 leader_election.cc:288] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27cb5473eeab4ac2b5f476491539141a (127.29.78.130:46011), df117490ee2743b8a4b7922090969281 (127.29.78.129:43377)
I20260810 22:58:08.438786 10739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.432146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46011 (local address 127.29.78.131:60559)
0810 22:58:08.433196 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.433213 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:08.433328 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:08.434223 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:08.434226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:08.434246 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:08.434484 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.434493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.435575 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.435578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.436984 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.436994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.438164 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.438167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.438386 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.438412 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:58:08.438510 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 22:58:08.438605 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":71}
I20260810 22:58:08.439028 10753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.433979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:59179 (local address 127.29.78.129:43377)
0810 22:58:08.434108 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.434112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.434128 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:58:08.434307 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.434311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.434381 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.434447 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:08.435017 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.435756 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.437591 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.438274 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.438379 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.438422 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.438832 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.438910 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:58:08.439283 10756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.433121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60559 (local address 127.29.78.130:46011)
0810 22:58:08.433258 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.433262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.433301 (+ 39us) server_negotiation.cc:407] Connection header received
0810 22:58:08.433477 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.433481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.433538 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.433621 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:08.434668 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.435407 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.437268 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.437899 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.438010 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.438065 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.439095 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.439171 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:58:08.439684 10768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.432466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43377 (local address 127.29.78.131:59179)
0810 22:58:08.433971 (+ 1505us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.433985 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:08.434101 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:08.434622 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:08.434625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:08.434645 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:08.434881 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.434892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.435922 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.435925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.437428 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:08.437439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.438515 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.438518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:08.438692 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.438713 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:58:08.439321 (+ 608us) client_negotiation.cc:240] Negotiation successful
0810 22:58:08.439395 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1441,"mutex_wait_us":3,"thread_start_us":276,"threads_started":1}
I20260810 22:58:08.440377 10527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73082685fc694096af03af2ac083fda7" candidate_uuid: "caaad23d2a704b4192a8e3bed68510bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27cb5473eeab4ac2b5f476491539141a" is_pre_election: true
I20260810 22:58:08.440692 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73082685fc694096af03af2ac083fda7" candidate_uuid: "caaad23d2a704b4192a8e3bed68510bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df117490ee2743b8a4b7922090969281" is_pre_election: true
I20260810 22:58:08.440699 10527 raft_consensus.cc:2454] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caaad23d2a704b4192a8e3bed68510bc in term 0.
I20260810 22:58:08.441018 10362 raft_consensus.cc:2454] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caaad23d2a704b4192a8e3bed68510bc in term 0.
I20260810 22:58:08.441377 10609 leader_election.cc:302] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cb5473eeab4ac2b5f476491539141a, caaad23d2a704b4192a8e3bed68510bc; no voters:
I20260810 22:58:08.441907 10765 raft_consensus.cc:2781] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:58:08.441998 10765 raft_consensus.cc:482] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:58:08.442040 10765 raft_consensus.cc:3037] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:08.443859 10765 raft_consensus.cc:504] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.444211 10765 leader_election.cc:288] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [CANDIDATE]: Term 1 election: Requested vote from peers 27cb5473eeab4ac2b5f476491539141a (127.29.78.130:46011), df117490ee2743b8a4b7922090969281 (127.29.78.129:43377)
I20260810 22:58:08.444947 10527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73082685fc694096af03af2ac083fda7" candidate_uuid: "caaad23d2a704b4192a8e3bed68510bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27cb5473eeab4ac2b5f476491539141a"
I20260810 22:58:08.445061 10527 raft_consensus.cc:3037] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:08.446591 10527 raft_consensus.cc:2454] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caaad23d2a704b4192a8e3bed68510bc in term 1.
I20260810 22:58:08.446969 10609 leader_election.cc:302] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27cb5473eeab4ac2b5f476491539141a, caaad23d2a704b4192a8e3bed68510bc; no voters:
I20260810 22:58:08.447232 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73082685fc694096af03af2ac083fda7" candidate_uuid: "caaad23d2a704b4192a8e3bed68510bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df117490ee2743b8a4b7922090969281"
I20260810 22:58:08.447364 10362 raft_consensus.cc:3037] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:08.449080 10362 raft_consensus.cc:2454] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caaad23d2a704b4192a8e3bed68510bc in term 1.
I20260810 22:58:08.449357 10765 raft_consensus.cc:2781] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:58:08.449661 10765 raft_consensus.cc:686] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 1 LEADER]: Becoming Leader. State: Replica: caaad23d2a704b4192a8e3bed68510bc, State: Running, Role: LEADER
I20260810 22:58:08.450153 10765 consensus_queue.cc:237] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } }
I20260810 22:58:08.453843 10183 catalog_manager.cc:5495] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc reported cstate change: term changed from 0 to 1, leader changed from <none> to caaad23d2a704b4192a8e3bed68510bc (127.29.78.131). New cstate: current_term: 1 leader_uuid: "caaad23d2a704b4192a8e3bed68510bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caaad23d2a704b4192a8e3bed68510bc" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 46185 } health_report { overall_health: HEALTHY } } }
I20260810 22:58:08.487545 10755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.479261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.29.78.131:46185)
0810 22:58:08.482018 (+ 2757us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:08.482024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:08.482043 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:58:08.482137 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:08.482140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:08.482213 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:08.482353 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:58:08.482762 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.483590 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.484930 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:08.485272 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:08.485437 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:08.485990 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:58:08.486577 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:58:08.486695 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:08.487215 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 22:58:08.487278 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2617}
I20260810 22:58:08.496126 10362 raft_consensus.cc:1261] T 73082685fc694096af03af2ac083fda7 P df117490ee2743b8a4b7922090969281 [term 1 FOLLOWER]: Refusing update from remote peer caaad23d2a704b4192a8e3bed68510bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:58:08.497195 10769 consensus_queue.cc:1035] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "df117490ee2743b8a4b7922090969281" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:58:08.504906 10527 raft_consensus.cc:1261] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 1 FOLLOWER]: Refusing update from remote peer caaad23d2a704b4192a8e3bed68510bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:58:08.505659 10769 consensus_queue.cc:1035] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "27cb5473eeab4ac2b5f476491539141a" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:58:08.516461 10772 mvcc.cc:205] Tried to move back new op lower bound from 7317105412069785600 to 7317105411894972416. Current Snapshot: MvccSnapshot[applied={T|T < 7317105412069785600}]
I20260810 22:58:08.519476 10773 mvcc.cc:205] Tried to move back new op lower bound from 7317105412069785600 to 7317105411894972416. Current Snapshot: MvccSnapshot[applied={T|T < 7317105412069785600 or (T in {7317105412069785600})}]
I20260810 22:58:08.527227 10775 mvcc.cc:205] Tried to move back new op lower bound from 7317105412069785600 to 7317105411894972416. Current Snapshot: MvccSnapshot[applied={T|T < 7317105412069785600}]
I20260810 22:58:08.531788 10183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:58:08.531937 10183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:58:08.535880 10183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d280dbc8adc42fd928aa7ebec11908f: Sending DeleteTablet for 3 replicas of tablet 73082685fc694096af03af2ac083fda7
I20260810 22:58:08.536706 10352 tablet_service.cc:1514] Processing DeleteTablet for tablet 73082685fc694096af03af2ac083fda7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:58:08 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:39156
I20260810 22:58:08.537851 10501 tablet_service.cc:1514] Processing DeleteTablet for tablet 73082685fc694096af03af2ac083fda7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:58:08 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:55426
I20260810 22:58:08.538678 10643 tablet_service.cc:1514] Processing DeleteTablet for tablet 73082685fc694096af03af2ac083fda7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:58:08 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57518
I20260810 22:58:08.539688 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 10274
I20260810 22:58:08.551319 10784 tablet_replica.cc:330] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: stopping tablet replica
I20260810 22:58:08.551724 10784 raft_consensus.cc:2229] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:58:08.552186 10784 raft_consensus.cc:2258] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:58:08.552881 10783 tablet_replica.cc:330] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a: stopping tablet replica
I20260810 22:58:08.553274 10783 raft_consensus.cc:2229] T 73082685fc694096af03af2ac083fda7 P 27cb5473eeab4ac2b5f476491539141a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:58:08.554875 10784 ts_tablet_manager.cc:1905] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 22:58:08.558074 10168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:43377 (error 108)
I20260810 22:58:08.559942 10784 ts_tablet_manager.cc:1918] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:58:08.560209 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 10437
I20260810 22:58:08.571058 10784 log.cc:1196] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/73082685fc694096af03af2ac083fda7
I20260810 22:58:08.571517 10784 ts_tablet_manager.cc:1939] T 73082685fc694096af03af2ac083fda7 P caaad23d2a704b4192a8e3bed68510bc: Deleting consensus metadata
I20260810 22:58:08.573024 10169 catalog_manager.cc:4841] TS caaad23d2a704b4192a8e3bed68510bc (127.29.78.131:46185): tablet 73082685fc694096af03af2ac083fda7 (table test-table [id=00f77f4b988a4264850e6087693ca23d]) successfully deleted
I20260810 22:58:08.574290 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.573936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43377 (local address 127.0.0.1:39170)
0810 22:58:08.574088 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.574197 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:43377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 22:58:08.574566 10168 catalog_manager.cc:4584] TS df117490ee2743b8a4b7922090969281 (127.29.78.129:43377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73082685fc694096af03af2ac083fda7: Network error: Client connection negotiation failed: client connection to 127.29.78.129:43377: connect: Connection refused (error 111)
W20260810 22:58:08.587293 10749 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:58:08.586686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46011 (local address 127.0.0.1:55438)
0810 22:58:08.586833 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.586849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:08.586991 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:08.587182 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:46011: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 22:58:08.590797 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 10590
I20260810 22:58:08.628144 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.627736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43377 (local address 127.0.0.1:39186)
0810 22:58:08.627976 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.628047 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:43377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260810 22:58:08.629136 10749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:08.628896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43377 (local address 127.0.0.1:39200)
0810 22:58:08.629025 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:08.629072 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:43377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 22:58:08.631387 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 10149
I20260810 22:58:08.640771 30010 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 22:58:08.985044 30010 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 22:58:08.985229 30010 mini_ranger.cc:66] Stopping Ranger...
I20260810 22:58:09.329784 30010 mini_ranger.cc:68] Stopped Ranger
2026-08-10 22:58:09.330 UTC [9686] LOG: received smart shutdown request
2026-08-10 22:58:09.334 UTC [9686] LOG: background worker "logical replication launcher" (PID 9697) exited with exit code 1
2026-08-10 22:58:09.334 UTC [9690] LOG: shutting down
2026-08-10 22:58:09.494 UTC [9686] LOG: database system is shut down
2026-08-10T22:58:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46271 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:58:09 dist-test-slave-6nmv krb5kdc[10794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:58:09 dist-test-slave-6nmv krb5kdc[10794](info): set up 2 sockets
Aug 10 22:58:09 dist-test-slave-6nmv krb5kdc[10794](info): commencing operation
krb5kdc: starting...
W20260810 22:58:10.675840 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:58:10 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:58:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/rangeradmin_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/rangerlookup_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
I20260810 22:58:10.833988 30010 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskX35aNT/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/postgres -l logfile start
2026-08-10 22:58:13.373 UTC [10829] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 22:58:13.373 UTC [10829] LOG: listening on IPv4 address "127.29.78.148", port 40853
2026-08-10 22:58:13.377 UTC [10829] LOG: listening on Unix socket "/tmp/.s.PGSQL.40853"
2026-08-10 22:58:13.434 UTC [10832] LOG: database system was shut down at 2026-08-10 22:58:11 UTC
2026-08-10 22:58:13.439 UTC [10829] LOG: database system is ready to accept connections
I20260810 22:58:14.478700 30010 mini_postgres.cc:96] Postgres bound to 40853
2026-08-10 22:58:14.487 UTC [10841] FATAL: database "slave" does not exist
127.29.78.148:40853 - accepting connections
I20260810 22:58:14.487758 30010 mini_ranger.cc:161] Starting Ranger...
I20260810 22:58:14.505357 30010 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 22:58:14.977708 30010 mini_ranger.cc:87] Created ranger Postgres database
I20260810 22:58:14.977777 30010 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-admin
2026-08-10 22:58:15,262 [I] DB FLAVOR :POSTGRES
2026-08-10 22:58:15,263 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:58:15,263 [I] Checking connection..
2026-08-10 22:58:15,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 22:58:15,514 [I] Checking connection passed.
2026-08-10 22:58:15,515 [I] --------- Verifying version history table ---------
2026-08-10 22:58:15,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:58:15,770 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:58:15,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:58:16,028 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 22:58:16,028 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 22:58:16,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 22:58:16.256 UTC [10913] WARNING: there is no transaction in progress
2026-08-10 22:58:16,263 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 22:58:16,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 22:58:16,516 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 22:58:16,516 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:58:16,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 22:58:16,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 22:58:16,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6nmv.c.gcp-upstream.internal','N') ;"
2026-08-10 22:58:17,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:58:17,481 [I] Table x_portal_user does not exist in database ranger
2026-08-10 22:58:17,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:58:17,702 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 22:58:17,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:58:17,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:58:18,197 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 22:58:18,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 22:58:19.105 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.122 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.138 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.152 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.167 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.184 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.237 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.243 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.255 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.264 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.273 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.286 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.295 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.301 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.308 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19.314 UTC [11032] WARNING: there is no transaction in progress
2026-08-10 22:58:19,899 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 22:58:19,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 22:58:20,144 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 22:58:20,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 22:58:20,374 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 22:58:20,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 22:58:20,623 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 22:58:20,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 22:58:20,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 22:58:21,118 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6nmv.c.gcp-upstream.internal';"
2026-08-10 22:58:21,342 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 22:58:21,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 22:58:21,579 [I] DB_PATCHES have already been applied
I20260810 22:58:21.584589 30010 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-admin:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskX35aNT/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 22:58:21.584648 30010 mini_ranger.cc:193] Using host: 127.29.78.148
I20260810 22:58:21.588428 30010 mini_ranger.cc:227] Ranger admin URL: http://127.29.78.148:53851
Aug 10, 2026 10:58:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:58:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:58:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 10:58:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 10:58:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 10:58:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 10:58:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 22:58:22.711210 30010 mini_ranger.cc:160] Time spent starting Ranger: real 8.223s user 0.003s sys 0.008s
Aug 10, 2026 10:58:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 10:58:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 10:58:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 10:58:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 10:58:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 22:58:44.375059 30010 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.29.78.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.29.78.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.29.78.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/rangerkms_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.29.78.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.29.78.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.29.78.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/HTTP_127.29.78.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 22:58:44 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402724, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 22:58:44.463901 30010 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 22:58:44.474740 30010 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 22:58:45.339293 30010 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 22:58:45.339409 30010 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms
I20260810 22:58:45.339432 30010 mini_ranger_kms.cc:227] Using postgres at 127.29.78.148:40853
2026-08-10 22:58:45,535 [I] DB FLAVOR :POSTGRES
2026-08-10 22:58:45,535 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 22:58:45,535 [I] Checking connection
2026-08-10 22:58:45,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 22:58:45,778 [I] connection success
2026-08-10 22:58:45,778 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 22:58:45,778 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 22:58:45,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 22:58:46,031 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 22:58:46,031 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 22:58:46,031 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 22:58:46,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.29.78.148:40853/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 22:58:46,289 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 22:58:46.728399 30010 mini_ranger_kms.cc:310] Created kms service
I20260810 22:58:46.907011 30010 mini_ranger_kms.cc:326] Created kudu user
I20260810 22:58:46.953138 30010 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 22:58:46 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402724, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:47.348590 30010 mini_ranger_kms.cc:384] Added ranger policy
I20260810 22:58:47.348853 30010 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms:/tmp/dist-test-taskX35aNT/build/debug/bin/postgresql.jar:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskX35aNT/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 22:58:47.348892 30010 mini_ranger_kms.cc:240] Using host: 127.29.78.148
I20260810 22:58:47.353267 30010 mini_ranger_kms.cc:276] Ranger KMS PID: 11236
I20260810 22:58:47.353358 30010 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.29.78.148:41689
Aug 10, 2026 10:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 10:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 10:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 10:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 10:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 10:58:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 22:58:49 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786402729, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:49.661181 30010 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.197s user 0.000s sys 0.182s
I20260810 22:58:49.661311 30010 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 22:58:49.661387 30010 mini_ranger_kms.cc:401] 127.29.78.148:41689/kms/v1/keys
Aug 10 22:58:50 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402729, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 10 22:58:50 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402729, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.29.78.148@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
Aug 10 22:58:52 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:58:52.652650 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43805
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43805
--ranger_config_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:52.760871 11304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:52.761129 11304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:52.761166 11304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:52.765563 11304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:58:52.765625 11304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:52.765646 11304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:52.765664 11304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:58:52.765681 11304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:58:52.770987 11304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
--ranger_kms_url=127.29.78.148:41689/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43805
--ranger_config_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:52.772038 11304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:52.773028 11304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:52.781929 11309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:52.781904 11310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:52.781982 11304 server_base.cc:1034] running on GCE node
W20260810 22:58:52.781888 11312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:52.782622 11304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:52.783643 11304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:52.784824 11304 hybrid_clock.cc:648] HybridClock initialized: now 1786402732784801 us; error 38 us; skew 500 ppm
Aug 10 22:58:52 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402732, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:52.791321 11304 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:58:52.792171 11304 webserver.cc:460] Webserver started at http://127.29.78.190:39113/ using document root <none> and password file <none>
I20260810 22:58:52.792696 11304 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:52.792788 11304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:52.793043 11304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:52 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402732, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:52.847663 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd"
format_stamp: "Formatted at 2026-08-10 22:58:52 on dist-test-slave-6nmv"
server_key: "9136b0a7f4faf362f3f015465b719d48"
server_key_iv: "b7854fd96299e84669b453d5b2a48b08"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:52.848496 11304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd"
format_stamp: "Formatted at 2026-08-10 22:58:52 on dist-test-slave-6nmv"
server_key: "9136b0a7f4faf362f3f015465b719d48"
server_key_iv: "b7854fd96299e84669b453d5b2a48b08"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:52.853209 11304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:58:52.867607 11319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:52.869066 11304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:58:52.869199 11304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd"
format_stamp: "Formatted at 2026-08-10 22:58:52 on dist-test-slave-6nmv"
server_key: "9136b0a7f4faf362f3f015465b719d48"
server_key_iv: "b7854fd96299e84669b453d5b2a48b08"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:52.869324 11304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:52.895512 11304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:52.898337 11304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:52.898567 11304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:52.906878 11304 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43805
I20260810 22:58:52.906919 11378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43805 every 8 connection(s)
I20260810 22:58:52.907859 11304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:58:52.909658 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 11304
I20260810 22:58:52.909780 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:58:52.910075 30010 external_mini_cluster.cc:1428] Setting key bb1c9a8dded0d948d9da3f6c715bb762
I20260810 22:58:52.910827 11379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:52.917132 11379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Bootstrap starting.
Aug 10 22:58:52 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:52.920133 11379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:52.921717 11379 log.cc:824] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:52.924098 11379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: No bootstrap required, opened a new log
I20260810 22:58:52.924849 11382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:52.911406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.29.78.190:43805)
0810 22:58:52.911799 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:52.911815 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:52.911851 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:58:52.912483 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:52.912498 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:52.912802 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:52.913081 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:52.914021 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:52.914987 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:52.916215 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:52.916659 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:52.916768 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:52.919827 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:52.919834 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:52.919849 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:52.919882 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:52.921964 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:52.922432 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:52.922437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:52.922441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:52.922550 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:52.922850 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:52.922854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:52.922855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:52.923315 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:52.923519 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:52.923731 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 22:58:52.923999 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
I20260810 22:58:52.927609 11379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:52.927908 11379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:52.927981 11379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ffc03f9cd84a0ca6f6f925e8efcbfd, State: Initialized, Role: FOLLOWER
I20260810 22:58:52.928328 11379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:52.928450 11379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:58:52.928493 11379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:58:52.928555 11379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:52.929292 11379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:52.929677 11379 leader_election.cc:302] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ffc03f9cd84a0ca6f6f925e8efcbfd; no voters:
I20260810 22:58:52.930017 11379 leader_election.cc:288] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:58:52.930130 11384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:58:52.930385 11384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 LEADER]: Becoming Leader. State: Replica: 34ffc03f9cd84a0ca6f6f925e8efcbfd, State: Running, Role: LEADER
I20260810 22:58:52.930751 11384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:52.931237 11379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:58:52.932602 11385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } } }
I20260810 22:58:52.932727 11385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: This master's current role is: LEADER
I20260810 22:58:52.933100 11395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:58:52.933111 11386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ffc03f9cd84a0ca6f6f925e8efcbfd. Latest consensus state: current_term: 1 leader_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } } }
I20260810 22:58:52.933336 11386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: This master's current role is: LEADER
I20260810 22:58:52.935886 11395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:58:52.936296 11379 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:58:52.944869 11395 catalog_manager.cc:1347] Generated new cluster ID: 544cae4bd8e44c9b8ca0cf67619b7e0d
I20260810 22:58:52.944938 11395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:58:52.964677 11395 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:58:52.966084 11395 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:58:52.975373 11395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Generated new TSK 0
I20260810 22:58:52.976029 11395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:58:53.692879 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43805
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:53.846119 11430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:53.846449 11430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:53.846509 11430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:53.852094 11430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:53.852159 11430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:53.852242 11430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:58:53.859668 11430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.29.78.148:41689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:53.860857 11430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:53.862107 11430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:53.873894 11440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:53.874058 11441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:53.874250 11443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:53.874343 11430 server_base.cc:1034] running on GCE node
I20260810 22:58:53.874743 11430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:53.875439 11430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:53.876652 11430 hybrid_clock.cc:648] HybridClock initialized: now 1786402733876632 us; error 33 us; skew 500 ppm
Aug 10 22:58:53 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402733, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:53.880388 11430 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:58:53.881351 11430 webserver.cc:460] Webserver started at http://127.29.78.129:33277/ using document root <none> and password file <none>
I20260810 22:58:53.881975 11430 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:53.882066 11430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:53.882302 11430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:53 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402733, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:53.906315 11430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "fb1ce61bf4704eb496d6b178061c6a9b"
format_stamp: "Formatted at 2026-08-10 22:58:53 on dist-test-slave-6nmv"
server_key: "71b5628e443048d618cd9f54971235b5"
server_key_iv: "1998af461e7f45d429723362ce1df969"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:53.907104 11430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "fb1ce61bf4704eb496d6b178061c6a9b"
format_stamp: "Formatted at 2026-08-10 22:58:53 on dist-test-slave-6nmv"
server_key: "71b5628e443048d618cd9f54971235b5"
server_key_iv: "1998af461e7f45d429723362ce1df969"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:53.921103 11430 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.000s
I20260810 22:58:53.941326 11451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:53.942736 11430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:58:53.942874 11430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "fb1ce61bf4704eb496d6b178061c6a9b"
format_stamp: "Formatted at 2026-08-10 22:58:53 on dist-test-slave-6nmv"
server_key: "71b5628e443048d618cd9f54971235b5"
server_key_iv: "1998af461e7f45d429723362ce1df969"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:53.943018 11430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:53.974010 11430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:53.978184 11430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:53.978811 11430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:53.979619 11430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:53.980826 11430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:53.980950 11430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:53.981081 11430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:53.981176 11430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:53.995877 11430 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:40685
I20260810 22:58:53.996029 11580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:40685 every 8 connection(s)
I20260810 22:58:53.997097 11430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:58:54.004796 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 11430
I20260810 22:58:54.004949 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:58:54.005307 30010 external_mini_cluster.cc:1428] Setting key 5b9f48a46e1a62fc32e7b57ebd381f9f
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402733, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:54.014423 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:53.999304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:45809 (local address 127.29.78.190:43805)
0810 22:58:53.999571 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:53.999575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:54.000442 (+ 867us) server_negotiation.cc:407] Connection header received
0810 22:58:54.001707 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:54.001711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:54.001777 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:54.001904 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:54.003212 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.004064 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.006078 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.006485 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.006567 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:54.009766 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:54.009770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:54.009773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:54.009801 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:54.011781 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:54.012413 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:54.012416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:54.012419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:54.012531 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:54.013054 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:54.013057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:54.013059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:54.013289 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:54.013365 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:54.014104 (+ 739us) server_negotiation.cc:299] Negotiation successful
0810 22:58:54.014239 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":81,"threads_started":1}
I20260810 22:58:54.016736 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:53.999746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43805 (local address 127.29.78.129:45809)
0810 22:58:54.000285 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:54.000339 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:54.001392 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:54.002062 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:54.002070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:54.002391 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:54.003038 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:54.003057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.004213 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.004217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:54.005695 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:54.005706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.006778 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.006781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:54.007007 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:54.007726 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:54.007747 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:54.009542 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:54.011920 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:54.011926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:54.011944 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:54.012290 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:54.012652 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:54.012655 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:54.012657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:54.012936 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:54.013449 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:54.013467 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:54.013783 (+ 316us) client_negotiation.cc:769] Sending connection context
0810 22:58:54.015205 (+ 1422us) client_negotiation.cc:240] Negotiation successful
0810 22:58:54.015592 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:58:54.018651 11581 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43805
I20260810 22:58:54.018900 11581 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:54.019480 11581 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:54.021382 11336 ts_manager.cc:194] Registered new tserver with Master: fb1ce61bf4704eb496d6b178061c6a9b (127.29.78.129:40685)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 22:58:54.022907 11336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:45809
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:58:54.041875 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43805
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:54.193063 11589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:54.193384 11589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:54.193441 11589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:54.199124 11589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:54.199231 11589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:54.199323 11589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:58:54.206699 11589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.29.78.148:41689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:54.208094 11589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:54.209450 11589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:54.223380 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:54.223858 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:54.224138 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:54.225215 11589 server_base.cc:1034] running on GCE node
I20260810 22:58:54.225718 11589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:54.226619 11589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:54.227798 11589 hybrid_clock.cc:648] HybridClock initialized: now 1786402734227735 us; error 93 us; skew 500 ppm
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:54.231822 11589 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:58:54.233160 11589 webserver.cc:460] Webserver started at http://127.29.78.130:42907/ using document root <none> and password file <none>
I20260810 22:58:54.234071 11589 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:54.234182 11589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:54.234438 11589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:54.252146 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "12be0fcf1c1943249632f75e9d9b89ab"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "994f95084dcbb99d5f5c9465132f6bfa"
server_key_iv: "8fcfad412f2a240052bc3c5fb793369e"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:54.253185 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "12be0fcf1c1943249632f75e9d9b89ab"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "994f95084dcbb99d5f5c9465132f6bfa"
server_key_iv: "8fcfad412f2a240052bc3c5fb793369e"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:54.265931 11589 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260810 22:58:54.283689 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:54.285789 11589 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 22:58:54.285993 11589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "12be0fcf1c1943249632f75e9d9b89ab"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "994f95084dcbb99d5f5c9465132f6bfa"
server_key_iv: "8fcfad412f2a240052bc3c5fb793369e"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:54.286168 11589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:54.319770 11589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:54.323453 11589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:54.323675 11589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:54.324378 11589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:54.325438 11589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:54.325510 11589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:54.325577 11589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:54.325608 11589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:54.337805 11589 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:43163
I20260810 22:58:54.338214 11730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:43163 every 8 connection(s)
I20260810 22:58:54.338873 11589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:58:54.346410 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 11589
I20260810 22:58:54.346550 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:58:54.346873 30010 external_mini_cluster.cc:1428] Setting key b365bf2267e193b77576be4f390541d0
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:58:54.360780 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:54.342241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49115 (local address 127.29.78.190:43805)
0810 22:58:54.342387 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:54.342392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:54.343337 (+ 945us) server_negotiation.cc:407] Connection header received
0810 22:58:54.344382 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:54.344385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:54.344440 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:54.344542 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:54.345723 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.346348 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.347968 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.348276 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.348379 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:54.351979 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:54.351985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:54.351990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:54.352025 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:54.353989 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:54.358665 (+ 4676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:54.358671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:54.358675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:54.358780 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:54.359425 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:54.359429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:54.359431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:54.359693 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:54.359766 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:54.360449 (+ 683us) server_negotiation.cc:299] Negotiation successful
0810 22:58:54.360601 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:58:54.361801 11733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:54.341499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43805 (local address 127.29.78.130:49115)
0810 22:58:54.343146 (+ 1647us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:54.343186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:54.344158 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:54.344715 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:54.344721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:54.344991 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:54.345585 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:54.345598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.346540 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.346550 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:54.347814 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:54.347824 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.348557 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.348561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:54.348837 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:54.349779 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:54.349805 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:54.351755 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:54.358065 (+ 6310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:54.358071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:54.358091 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:54.358503 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:54.358907 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:54.358910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:54.358912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:54.359292 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:54.359823 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:54.359828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:54.360119 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 22:58:54.360392 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 22:58:54.360726 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1430,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260810 22:58:54.363178 11731 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43805
I20260810 22:58:54.363421 11731 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:54.364037 11731 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:54.365075 11336 ts_manager.cc:194] Registered new tserver with Master: 12be0fcf1c1943249632f75e9d9b89ab (127.29.78.130:43163)
I20260810 22:58:54.365741 11336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:49115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:58:54.380837 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43805
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
W20260810 22:58:54.524418 11738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:54.524811 11738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:54.524902 11738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:54.530563 11738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:54.530709 11738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:54.530835 11738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:58:54.538480 11738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.29.78.148:41689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:54.540045 11738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:54.541388 11738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:54.554193 11744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:54.555209 11743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:54.555576 11738 server_base.cc:1034] running on GCE node
W20260810 22:58:54.555523 11746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:54.556195 11738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:54.556869 11738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:54.558048 11738 hybrid_clock.cc:648] HybridClock initialized: now 1786402734557976 us; error 84 us; skew 500 ppm
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:54.561574 11738 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:58:54.562582 11738 webserver.cc:460] Webserver started at http://127.29.78.131:33471/ using document root <none> and password file <none>
I20260810 22:58:54.563233 11738 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:54.563306 11738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:54.563589 11738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:54.600059 11738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "98ea7859ace84198923d82a57e32849b"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "dff4a38f07237df8234f7019edbc040e"
server_key_iv: "761db64886fd41eb845805682af243fe"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:54.607453 11738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "98ea7859ace84198923d82a57e32849b"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "dff4a38f07237df8234f7019edbc040e"
server_key_iv: "761db64886fd41eb845805682af243fe"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:54.623413 11738 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.004s sys 0.000s
I20260810 22:58:54.662235 11754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:54.664016 11738 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.002s sys 0.000s
I20260810 22:58:54.664155 11738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "98ea7859ace84198923d82a57e32849b"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "dff4a38f07237df8234f7019edbc040e"
server_key_iv: "761db64886fd41eb845805682af243fe"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:54.664299 11738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:54.692625 11738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:54.695946 11738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:54.696182 11738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:54.696817 11738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:54.697822 11738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:54.697947 11738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:54.698060 11738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:54.698132 11738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:54.710695 11738 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41517
I20260810 22:58:54.711827 11738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:58:54.714186 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 11738
I20260810 22:58:54.714361 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:58:54.714731 30010 external_mini_cluster.cc:1428] Setting key f5de89a52d0957d209655a33c7962e24
I20260810 22:58:54.712610 11880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41517 every 8 connection(s)
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:54.730036 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:54.717022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:50273 (local address 127.29.78.190:43805)
0810 22:58:54.717134 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:54.717138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:54.717615 (+ 477us) server_negotiation.cc:407] Connection header received
0810 22:58:54.718821 (+ 1206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:54.718824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:54.718874 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:54.718921 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:54.720437 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.721246 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.722928 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.723236 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.723300 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:54.726084 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:54.726087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:54.726090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:54.726114 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:54.727763 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:54.728382 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:54.728386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:54.728388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:54.728474 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:54.728855 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:54.728858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:54.728860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:54.729064 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:54.729164 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:54.729747 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 22:58:54.729879 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:58:54.731631 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:54.715895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43805 (local address 127.29.78.131:50273)
0810 22:58:54.717469 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:54.717523 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:54.718598 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:54.719313 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:54.719320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:54.719642 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:54.720277 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:54.720294 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.721390 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.721393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:54.722791 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:54.722802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:54.723472 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:54.723475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:54.723641 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:54.724285 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:54.724302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:54.725905 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:54.727907 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:54.727912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:54.727926 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:54.728262 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:54.728586 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:54.728588 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:54.728590 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:54.728755 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:54.729226 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:54.729231 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:54.729494 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 22:58:54.730408 (+ 914us) client_negotiation.cc:240] Negotiation successful
0810 22:58:54.730723 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":200,"threads_started":1}
I20260810 22:58:54.733050 11881 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43805
I20260810 22:58:54.733268 11881 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:54.733769 11881 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:54.734772 11336 ts_manager.cc:194] Registered new tserver with Master: 98ea7859ace84198923d82a57e32849b (127.29.78.131:41517)
I20260810 22:58:54.735420 11336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:50273
I20260810 22:58:54.740031 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:58:54.740713 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 11430
I20260810 22:58:54.753465 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 11589
I20260810 22:58:54.767978 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 11738
I20260810 22:58:54.793066 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 11304
I20260810 22:58:54.803357 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43805
--webserver_interface=127.29.78.190
--webserver_port=39113
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43805
--ranger_config_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:54.907618 11884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:54.907860 11884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:54.907907 11884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:54.911537 11884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:58:54.911581 11884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:54.911612 11884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:54.911643 11884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:58:54.911659 11884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:58:54.916757 11884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43805
--ranger_config_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=39113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:54.917901 11884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:54.918915 11884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:54.927187 11890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:54.927191 11891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:54.927356 11884 server_base.cc:1034] running on GCE node
W20260810 22:58:54.927459 11893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:54.928081 11884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:54.929452 11884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:54.930639 11884 hybrid_clock.cc:648] HybridClock initialized: now 1786402734930610 us; error 39 us; skew 500 ppm
Aug 10 22:58:54 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:54.934232 11884 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:58:54.935261 11884 webserver.cc:460] Webserver started at http://127.29.78.190:39113/ using document root <none> and password file <none>
I20260810 22:58:54.935864 11884 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:54.935964 11884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402734, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:55.024837 11884 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 22:58:55.027508 11902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.028852 11884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 22:58:55.029000 11884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd"
format_stamp: "Formatted at 2026-08-10 22:58:52 on dist-test-slave-6nmv"
server_key: "9136b0a7f4faf362f3f015465b719d48"
server_key_iv: "b7854fd96299e84669b453d5b2a48b08"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:55.029554 11884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:55.048130 11884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:55.051251 11884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:55.051452 11884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:55.059132 11884 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43805
I20260810 22:58:55.059231 11962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43805 every 8 connection(s)
I20260810 22:58:55.060163 11884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:58:55.061514 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 11884
I20260810 22:58:55.062041 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:40685
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:43805
--webserver_port=33277
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:58:55.065086 11963 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:58:55.066560 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Bootstrap starting.
I20260810 22:58:55.085206 11963 log.cc:824] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:55.093466 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:58:55.093981 11963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Bootstrap complete.
I20260810 22:58:55.099550 11963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:55.100108 11963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ffc03f9cd84a0ca6f6f925e8efcbfd, State: Initialized, Role: FOLLOWER
I20260810 22:58:55.100742 11963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:55.100957 11963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:58:55.101046 11963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:58:55.101181 11963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:58:55.103750 11963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:55.104158 11963 leader_election.cc:302] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ffc03f9cd84a0ca6f6f925e8efcbfd; no voters:
I20260810 22:58:55.104560 11963 leader_election.cc:288] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:58:55.105939 11963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:58:55.108279 11963 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 22:58:55.109336 11967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:58:55.109575 11967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [term 2 LEADER]: Becoming Leader. State: Replica: 34ffc03f9cd84a0ca6f6f925e8efcbfd, State: Running, Role: LEADER
I20260810 22:58:55.109946 11967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } }
I20260810 22:58:55.110929 11967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } } }
I20260810 22:58:55.111075 11967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: This master's current role is: LEADER
I20260810 22:58:55.111294 11967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ffc03f9cd84a0ca6f6f925e8efcbfd. Latest consensus state: current_term: 2 leader_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ffc03f9cd84a0ca6f6f925e8efcbfd" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43805 } } }
I20260810 22:58:55.111405 11967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd [sys.catalog]: This master's current role is: LEADER
I20260810 22:58:55.116118 11984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:58:55.120237 11984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:58:55.121165 11984 catalog_manager.cc:1259] Loaded cluster ID: 544cae4bd8e44c9b8ca0cf67619b7e0d
I20260810 22:58:55.121269 11984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:58:55.124574 11984 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:58:55.125336 11984 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Loaded TSK: 0
I20260810 22:58:55.126866 11984 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:58:55.219871 11965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:55.220643 11965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:55.220785 11965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:55.226424 11965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:55.226591 11965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:55.226744 11965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:58:55.234251 11965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:40685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=33277
--tserver_master_addrs=127.29.78.190:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:55.235646 11965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:55.236873 11965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:55.249080 11998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:55.249518 11997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:55.250316 11965 server_base.cc:1034] running on GCE node
W20260810 22:58:55.253993 12001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:55.254454 11965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:55.255141 11965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:55.262490 11965 hybrid_clock.cc:648] HybridClock initialized: now 1786402735262454 us; error 43 us; skew 500 ppm
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:55.267691 11965 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:58:55.268744 11965 webserver.cc:460] Webserver started at http://127.29.78.129:33277/ using document root <none> and password file <none>
I20260810 22:58:55.269408 11965 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:55.269510 11965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:55.304404 11965 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.003s sys 0.000s
I20260810 22:58:55.317862 12015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.319707 11965 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260810 22:58:55.319860 11965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "fb1ce61bf4704eb496d6b178061c6a9b"
format_stamp: "Formatted at 2026-08-10 22:58:53 on dist-test-slave-6nmv"
server_key: "71b5628e443048d618cd9f54971235b5"
server_key_iv: "1998af461e7f45d429723362ce1df969"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:55.320626 11965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:55.357709 11965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:55.361244 11965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:55.361479 11965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:55.362285 11965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:55.363312 11965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:55.363416 11965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.363547 11965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:55.363634 11965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.374053 11965 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:40685
I20260810 22:58:55.375362 11965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:58:55.377658 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 11965
I20260810 22:58:55.378209 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:43163
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:43805
--webserver_port=42907
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:58:55.382565 12138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:40685 every 8 connection(s)
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:55.413731 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:55.384724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43805 (local address 127.29.78.129:57543)
0810 22:58:55.398274 (+ 13550us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:55.398332 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:55.399333 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:55.400986 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:55.400994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:55.401300 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:55.401866 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:55.401884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.403462 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.403465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:55.404686 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:55.404696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.405718 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.405722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:55.405893 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:55.406565 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:55.406584 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:55.408258 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:55.410573 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:55.410578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:55.410589 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:55.410969 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:55.411316 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:55.411318 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:55.411320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:55.411474 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:55.412142 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:55.412147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:55.412388 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 22:58:55.412566 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 22:58:55.412860 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13328,"thread_start_us":143,"threads_started":1}
I20260810 22:58:55.415066 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:55.386102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:57543 (local address 127.29.78.190:43805)
0810 22:58:55.397962 (+ 11860us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:55.397985 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:55.399638 (+ 1653us) server_negotiation.cc:407] Connection header received
0810 22:58:55.399905 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:55.399916 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:55.400267 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:55.400572 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:55.402033 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.403320 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.404816 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.405398 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.405537 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:55.408471 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:55.408477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:55.408492 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:55.408521 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:55.410461 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.411105 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.411110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.411114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.411227 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.411567 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.411570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.411571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.411985 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:55.412095 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:55.414067 (+ 1972us) server_negotiation.cc:299] Negotiation successful
0810 22:58:55.414298 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11676,"thread_start_us":107,"threads_started":1}
I20260810 22:58:55.418356 12139 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43805
I20260810 22:58:55.418630 12139 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.419155 12139 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
W20260810 22:58:55.421183 12139 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:43805 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:58:55.421304 12139 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.421504 12139 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:55.422359 12139 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.422529 12139 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
W20260810 22:58:55.423264 12139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:58:55.540697 12141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:55.541100 12141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:55.541224 12141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:55.546188 12141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:55.546341 12141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:55.546492 12141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:58:55.553543 12141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:43163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=42907
--tserver_master_addrs=127.29.78.190:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:55.554786 12141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:55.556025 12141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:55.566476 12153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:55.566664 12151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:55.566586 12150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:55.567509 12141 server_base.cc:1034] running on GCE node
I20260810 22:58:55.567986 12141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:55.568776 12141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:55.569933 12141 hybrid_clock.cc:648] HybridClock initialized: now 1786402735569906 us; error 36 us; skew 500 ppm
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:55.573406 12141 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:58:55.574450 12141 webserver.cc:460] Webserver started at http://127.29.78.130:42907/ using document root <none> and password file <none>
I20260810 22:58:55.575119 12141 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:55.575228 12141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:55.594931 12141 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260810 22:58:55.598009 12163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.599295 12141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:58:55.599419 12141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "12be0fcf1c1943249632f75e9d9b89ab"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "994f95084dcbb99d5f5c9465132f6bfa"
server_key_iv: "8fcfad412f2a240052bc3c5fb793369e"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:55.600041 12141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:55.615434 12141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:55.618871 12141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:55.619071 12141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:55.619670 12141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:55.620466 12141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:55.620510 12141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.620579 12141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:55.620609 12141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.631289 12141 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:43163
I20260810 22:58:55.631424 12286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:43163 every 8 connection(s)
I20260810 22:58:55.632507 12141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:58:55.632817 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 12141
I20260810 22:58:55.633383 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.29.78.148:41689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:41517
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:43805
--webserver_port=33471
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:36513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:55.651986 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:55.634959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:60191 (local address 127.29.78.190:43805)
0810 22:58:55.635129 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:55.635136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:55.638289 (+ 3153us) server_negotiation.cc:407] Connection header received
0810 22:58:55.639471 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:55.639475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:55.639549 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:55.639647 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:55.640752 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.641291 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.642536 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.642783 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.642894 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:55.647777 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:55.647782 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:55.647785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:55.647814 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:55.649655 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.650382 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.650386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.650387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.650481 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.650811 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.650815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.650816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.650992 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:55.651091 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:55.651663 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 22:58:55.651802 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:58:55.653746 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:55.635387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43805 (local address 127.29.78.130:60191)
0810 22:58:55.638127 (+ 2740us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:55.638171 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:55.639144 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:55.639827 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:55.639834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:55.640110 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:55.640599 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:55.640612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.641456 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.641459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:55.642400 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:55.642409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.642936 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.642940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:55.643106 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:55.646002 (+ 2896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:55.646023 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:55.647574 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:55.649899 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:55.649905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:55.649920 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:55.650260 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:55.650569 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:55.650572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:55.650574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:55.650711 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:55.651091 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:55.651098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:55.651339 (+ 241us) client_negotiation.cc:769] Sending connection context
0810 22:58:55.652452 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0810 22:58:55.652792 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2550,"thread_start_us":147,"threads_started":1}
I20260810 22:58:55.654940 12287 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43805
I20260810 22:58:55.655171 12287 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.655711 12287 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
W20260810 22:58:55.657177 12287 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:43805 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:58:55.657306 12287 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.657497 12287 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:55.658291 12287 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.658489 12287 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
W20260810 22:58:55.659237 12287 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:58:55.783507 12290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:55.783775 12290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:55.783819 12290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:55.789332 12290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:55.789471 12290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:55.789603 12290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:58:55.795589 12290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:36513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.29.78.148:41689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:41517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=33471
--tserver_master_addrs=127.29.78.190:43805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:55.796761 12290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:55.798015 12290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:55.808856 12300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:55.808822 12298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:55.808830 12297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:55.810381 12290 server_base.cc:1034] running on GCE node
I20260810 22:58:55.810827 12290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:55.811563 12290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:55.812825 12290 hybrid_clock.cc:648] HybridClock initialized: now 1786402735812796 us; error 42 us; skew 500 ppm
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:55.816497 12290 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:58:55.817516 12290 webserver.cc:460] Webserver started at http://127.29.78.131:33471/ using document root <none> and password file <none>
I20260810 22:58:55.818225 12290 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:55.818346 12290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:55.836398 12290 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:58:55.842109 12310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.843430 12290 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 22:58:55.843580 12290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "98ea7859ace84198923d82a57e32849b"
format_stamp: "Formatted at 2026-08-10 22:58:54 on dist-test-slave-6nmv"
server_key: "dff4a38f07237df8234f7019edbc040e"
server_key_iv: "761db64886fd41eb845805682af243fe"
server_key_version: "kuduclusterkey@0"
I20260810 22:58:55.844156 12290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:55.862080 12290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:55.865737 12290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:55.866014 12290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:55.866741 12290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:58:55.867780 12290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:58:55.867864 12290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.867960 12290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:58:55.868012 12290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:55.880498 12290 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41517
I20260810 22:58:55.881850 12290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:58:55.881951 12429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41517 every 8 connection(s)
I20260810 22:58:55.887789 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 12290
Aug 10 22:58:55 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402735, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:55.899179 12145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:55.884070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:34949 (local address 127.29.78.190:43805)
0810 22:58:55.884213 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:55.884218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:55.885273 (+ 1055us) server_negotiation.cc:407] Connection header received
0810 22:58:55.886594 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:55.886598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:55.886669 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:55.886768 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:55.888199 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.889081 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.891038 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.891382 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.891475 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:55.894797 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:55.894802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:55.894804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:55.894852 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:55.896870 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.897348 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.897351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.897353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.897436 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.897883 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.897886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.897887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.898128 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:55.898227 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:55.898867 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 22:58:55.899012 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:58:55.899847 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:55.889358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.29.78.190:43805)
0810 22:58:55.889635 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:55.889639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:55.889652 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:58:55.889685 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:55.889687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:55.889738 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:55.889901 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:55.890650 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.891257 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.892529 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.892901 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.893010 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:55.894707 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:55.894712 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:55.894717 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:55.894745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:55.896576 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.897589 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.897591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.897593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.897659 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:55.898020 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:55.898023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:55.898025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:55.899015 (+ 990us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:55.899209 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:55.899612 (+ 403us) server_negotiation.cc:299] Negotiation successful
0810 22:58:55.899715 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"spinlock_wait_cycles":16256,"thread_start_us":112,"threads_started":1}
I20260810 22:58:55.900786 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:55.884542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43805 (local address 127.29.78.131:34949)
0810 22:58:55.885113 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:55.885165 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:55.886314 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:55.886983 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:55.886992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:55.887300 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:55.888022 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:55.888042 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.889567 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.889570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:55.890915 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:55.890927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:55.891515 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:55.891519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:55.891717 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:55.892384 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:55.892400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:55.894555 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:55.896985 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:55.896989 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:55.896999 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:55.897234 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:55.897568 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:55.897572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:55.897576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:55.897763 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:55.898283 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:55.898289 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:55.898571 (+ 282us) client_negotiation.cc:769] Sending connection context
0810 22:58:55.899386 (+ 815us) client_negotiation.cc:240] Negotiation successful
0810 22:58:55.899683 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":152,"threads_started":1}
I20260810 22:58:55.901765 12430 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43805
I20260810 22:58:55.902014 12430 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.902562 12430 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
W20260810 22:58:55.904316 12430 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:43805 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 22:58:55.904484 12430 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.904649 12430 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:55.905544 12430 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:55.905714 12430 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
W20260810 22:58:55.906524 12430 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 22:58:56 dist-test-slave-6nmv krb5kdc[10794](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786402736, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:56.423506 12139 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:56.423816 12139 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:56.425096 11912 ts_manager.cc:194] Registered new tserver with Master: fb1ce61bf4704eb496d6b178061c6a9b (127.29.78.129:40685)
I20260810 22:58:56.426904 11912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:57543
I20260810 22:58:56.659510 12287 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:56.659829 12287 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:56.660607 11912 ts_manager.cc:194] Registered new tserver with Master: 12be0fcf1c1943249632f75e9d9b89ab (127.29.78.130:43163)
I20260810 22:58:56.661268 11912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:60191
Aug 10 22:58:56 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402736, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:56.906731 12430 heartbeater.cc:464] Registering TS with master...
I20260810 22:58:56.907040 12430 heartbeater.cc:511] Master 127.29.78.190:43805 requested a full tablet report, sending...
I20260810 22:58:56.907855 11912 ts_manager.cc:194] Registered new tserver with Master: 98ea7859ace84198923d82a57e32849b (127.29.78.131:41517)
I20260810 22:58:56.908713 11912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:34949
I20260810 22:58:56.913089 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 22:58:56 dist-test-slave-6nmv krb5kdc[10794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402736, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for HTTP/127.29.78.148@KRBTEST.COM
I20260810 22:58:56.930251 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:56.914635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.29.78.190:43805)
0810 22:58:56.914912 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:56.914917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:56.918006 (+ 3089us) server_negotiation.cc:407] Connection header received
0810 22:58:56.918051 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:56.918054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:56.918102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:56.918182 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:56.919065 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.919865 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.920928 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.921261 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.921344 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:56.925709 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:56.925712 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:56.925715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:56.925740 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:56.927576 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:56.928079 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:56.928083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:56.928085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:56.928190 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:56.928513 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:56.928516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:56.928518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:56.928761 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:56.928902 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:56.929190 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 22:58:56.929333 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260810 22:58:56.934725 11912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:58:56.939394 11912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:58:56.974800 12464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:56.957751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:43163 (local address 127.0.0.1:55972)
0810 22:58:56.963728 (+ 5977us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:56.963741 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:56.963841 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:56.967733 (+ 3892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:56.967736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:56.967760 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:56.967874 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:56.967882 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.971074 (+ 3192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.971078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:56.972461 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:56.972474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.973895 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.973899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:56.974052 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:56.974080 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 22:58:56.974163 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 22:58:56.974240 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5903,"thread_start_us":369,"threads_started":1}
I20260810 22:58:56.975080 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:56.957974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55972 (local address 127.29.78.130:43163)
0810 22:58:56.966527 (+ 8553us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:56.966534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:56.966560 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:58:56.966636 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:56.966641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:56.967161 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:56.967310 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:56.969682 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.970917 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.972646 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.973547 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.973662 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:56.973804 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:56.974764 (+ 960us) server_negotiation.cc:299] Negotiation successful
0810 22:58:56.974865 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8129,"thread_start_us":90,"threads_started":1}
I20260810 22:58:56.978576 12200 tablet_service.cc:1467] Processing CreateTablet for tablet b3a4f3c2bbb74449afc2a7dc2728b072 (DEFAULT_TABLE table=test-table [id=690a721e1e5f464687217d9ddb69df1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:58:56.980185 12200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a4f3c2bbb74449afc2a7dc2728b072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:56.975526 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:56.957654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:41517 (local address 127.0.0.1:52370)
0810 22:58:56.962617 (+ 4963us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:56.962993 (+ 376us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:56.963119 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:56.965287 (+ 2168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:56.965293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:56.965325 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:56.965443 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:56.965449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.969497 (+ 4048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.969500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:56.970959 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:56.970971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.973838 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.973842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:56.973956 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:56.973997 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 22:58:56.974110 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 22:58:56.974230 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4777,"thread_start_us":739,"threads_started":1}
I20260810 22:58:56.984210 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:56.959094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40685 (local address 127.0.0.1:52634)
0810 22:58:56.974496 (+ 15402us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:56.974513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:56.974636 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:56.976147 (+ 1511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:56.976150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:56.976169 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:56.976263 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:56.976270 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.977820 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.977841 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:56.981984 (+ 4143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:56.982013 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.983793 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.983797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:56.983904 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:56.983928 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:58:56.983993 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 22:58:56.984077 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15335,"thread_start_us":145,"threads_started":1}
I20260810 22:58:56.984717 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:56.959570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.29.78.129:40685)
0810 22:58:56.975440 (+ 15870us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:56.975448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:56.975474 (+ 26us) server_negotiation.cc:407] Connection header received
0810 22:58:56.975548 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:56.975553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:56.975756 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:56.975911 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:56.976428 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.977663 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.982148 (+ 4485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.983075 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.983196 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:56.983326 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:56.984447 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0810 22:58:56.984545 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15741,"thread_start_us":74,"threads_started":1}
I20260810 22:58:56.986341 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:56.958099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52370 (local address 127.29.78.131:41517)
0810 22:58:56.964569 (+ 6470us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:56.964576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:56.964599 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:58:56.964684 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:56.964689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:56.964924 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:56.965057 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:56.968040 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.969320 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.971418 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:56.973102 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:56.973576 (+ 474us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:56.973747 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:56.986076 (+ 12329us) server_negotiation.cc:299] Negotiation successful
0810 22:58:56.986192 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6365,"thread_start_us":146,"threads_started":1}
I20260810 22:58:56.987794 12066 tablet_service.cc:1467] Processing CreateTablet for tablet b3a4f3c2bbb74449afc2a7dc2728b072 (DEFAULT_TABLE table=test-table [id=690a721e1e5f464687217d9ddb69df1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:58:56.988963 12066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a4f3c2bbb74449afc2a7dc2728b072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:56.991889 12360 tablet_service.cc:1467] Processing CreateTablet for tablet b3a4f3c2bbb74449afc2a7dc2728b072 (DEFAULT_TABLE table=test-table [id=690a721e1e5f464687217d9ddb69df1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:58:56.992973 12360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a4f3c2bbb74449afc2a7dc2728b072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:57.002231 12467 tablet_bootstrap.cc:492] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: Bootstrap starting.
I20260810 22:58:57.007035 12467 tablet_bootstrap.cc:654] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:57.008714 12467 log.cc:824] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:57.013562 12467 tablet_bootstrap.cc:492] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: No bootstrap required, opened a new log
I20260810 22:58:57.013777 12467 ts_tablet_manager.cc:1397] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260810 22:58:57.019747 12469 tablet_bootstrap.cc:492] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Bootstrap starting.
I20260810 22:58:57.019675 12467 raft_consensus.cc:348] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.020112 12467 raft_consensus.cc:374] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:57.020210 12467 raft_consensus.cc:729] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ea7859ace84198923d82a57e32849b, State: Initialized, Role: FOLLOWER
I20260810 22:58:57.020767 12467 consensus_queue.cc:260] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.022127 12467 ts_tablet_manager.cc:1428] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20260810 22:58:57.022543 12430 heartbeater.cc:503] Master 127.29.78.190:43805 was elected leader, sending a full tablet report...
I20260810 22:58:57.022662 12469 tablet_bootstrap.cc:654] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:57.023787 12469 log.cc:824] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:57.024262 12470 tablet_bootstrap.cc:492] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b: Bootstrap starting.
I20260810 22:58:57.027341 12470 tablet_bootstrap.cc:654] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b: Neither blocks nor log segments found. Creating new log.
I20260810 22:58:57.028910 12470 log.cc:824] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:58:57.032445 12469 tablet_bootstrap.cc:492] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: No bootstrap required, opened a new log
I20260810 22:58:57.032701 12469 ts_tablet_manager.cc:1397] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260810 22:58:57.033660 12470 tablet_bootstrap.cc:492] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b: No bootstrap required, opened a new log
I20260810 22:58:57.034024 12470 ts_tablet_manager.cc:1397] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260810 22:58:57.037775 12470 raft_consensus.cc:348] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.038082 12470 raft_consensus.cc:374] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:57.038148 12470 raft_consensus.cc:729] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1ce61bf4704eb496d6b178061c6a9b, State: Initialized, Role: FOLLOWER
I20260810 22:58:57.038167 12469 raft_consensus.cc:348] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.038512 12469 raft_consensus.cc:374] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:58:57.038601 12469 raft_consensus.cc:729] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12be0fcf1c1943249632f75e9d9b89ab, State: Initialized, Role: FOLLOWER
I20260810 22:58:57.039073 12469 consensus_queue.cc:260] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.039781 12287 heartbeater.cc:503] Master 127.29.78.190:43805 was elected leader, sending a full tablet report...
I20260810 22:58:57.040135 12469 ts_tablet_manager.cc:1428] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 22:58:57.040571 12470 consensus_queue.cc:260] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.041546 12470 ts_tablet_manager.cc:1428] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260810 22:58:57.041937 12139 heartbeater.cc:503] Master 127.29.78.190:43805 was elected leader, sending a full tablet report...
W20260810 22:58:57.133966 12431 tablet.cc:2369] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:58:57.140271 12288 tablet.cc:2369] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:58:57.140499 12140 tablet.cc:2369] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:58:57.153626 12471 raft_consensus.cc:482] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:58:57.153888 12471 raft_consensus.cc:504] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.155409 12471 leader_election.cc:288] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12be0fcf1c1943249632f75e9d9b89ab (127.29.78.130:43163), fb1ce61bf4704eb496d6b178061c6a9b (127.29.78.129:40685)
I20260810 22:58:57.161933 12460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:57.156727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:35637 (local address 127.29.78.130:43163)
0810 22:58:57.156883 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:57.156887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:57.157292 (+ 405us) server_negotiation.cc:407] Connection header received
0810 22:58:57.157511 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:57.157515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:57.157560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:57.157626 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:57.158163 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.158964 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.160529 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.161128 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.161223 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:57.161279 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:57.161673 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 22:58:57.161759 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 22:58:57.162464 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:57.155810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:43163 (local address 127.29.78.131:35637)
0810 22:58:57.157182 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:57.157196 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:57.157386 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:57.157761 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:57.157764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:57.157781 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:57.158023 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:57.158031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.159122 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.159125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:57.160382 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:57.160394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.161388 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.161391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:57.161499 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:57.161542 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 22:58:57.162228 (+ 686us) client_negotiation.cc:240] Negotiation successful
0810 22:58:57.162309 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1304,"thread_start_us":124,"threads_started":1}
I20260810 22:58:57.163378 12239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a4f3c2bbb74449afc2a7dc2728b072" candidate_uuid: "98ea7859ace84198923d82a57e32849b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12be0fcf1c1943249632f75e9d9b89ab" is_pre_election: true
I20260810 22:58:57.163729 12239 raft_consensus.cc:2454] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ea7859ace84198923d82a57e32849b in term 0.
I20260810 22:58:57.164218 12311 leader_election.cc:302] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12be0fcf1c1943249632f75e9d9b89ab, 98ea7859ace84198923d82a57e32849b; no voters:
I20260810 22:58:57.164505 12471 raft_consensus.cc:2781] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:58:57.164569 12471 raft_consensus.cc:482] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:58:57.164605 12471 raft_consensus.cc:3037] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:57.165943 12471 raft_consensus.cc:504] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.166285 12471 leader_election.cc:288] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [CANDIDATE]: Term 1 election: Requested vote from peers 12be0fcf1c1943249632f75e9d9b89ab (127.29.78.130:43163), fb1ce61bf4704eb496d6b178061c6a9b (127.29.78.129:40685)
I20260810 22:58:57.166869 12239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a4f3c2bbb74449afc2a7dc2728b072" candidate_uuid: "98ea7859ace84198923d82a57e32849b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12be0fcf1c1943249632f75e9d9b89ab"
I20260810 22:58:57.166999 12239 raft_consensus.cc:3037] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:57.167227 12465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:57.156880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:43125 (local address 127.29.78.129:40685)
0810 22:58:57.156992 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:57.156996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:57.163182 (+ 6186us) server_negotiation.cc:407] Connection header received
0810 22:58:57.163388 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:57.163391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:57.163443 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:57.163492 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:58:57.163960 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.164594 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.165985 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.166539 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.166632 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:57.166694 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:57.167027 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 22:58:57.167106 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:58:57.167539 12476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:57.156246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40685 (local address 127.29.78.131:43125)
0810 22:58:57.163106 (+ 6860us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:57.163120 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:57.163255 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:57.163681 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:57.163685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:57.163704 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:58:57.163774 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:57.163779 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.164727 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.164732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:57.165787 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:57.165799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.166775 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.166781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:57.166863 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:57.166902 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 22:58:57.167358 (+ 456us) client_negotiation.cc:240] Negotiation successful
0810 22:58:57.167427 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6800,"thread_start_us":93,"threads_started":1}
I20260810 22:58:57.168171 12239 raft_consensus.cc:2454] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98ea7859ace84198923d82a57e32849b in term 1.
I20260810 22:58:57.168505 12311 leader_election.cc:302] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12be0fcf1c1943249632f75e9d9b89ab, 98ea7859ace84198923d82a57e32849b; no voters:
I20260810 22:58:57.169090 12089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a4f3c2bbb74449afc2a7dc2728b072" candidate_uuid: "98ea7859ace84198923d82a57e32849b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" is_pre_election: true
I20260810 22:58:57.169101 12087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a4f3c2bbb74449afc2a7dc2728b072" candidate_uuid: "98ea7859ace84198923d82a57e32849b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1ce61bf4704eb496d6b178061c6a9b"
I20260810 22:58:57.169373 12089 raft_consensus.cc:2454] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98ea7859ace84198923d82a57e32849b in term 0.
I20260810 22:58:57.169610 12471 raft_consensus.cc:2781] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:58:57.169872 12471 raft_consensus.cc:686] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 1 LEADER]: Becoming Leader. State: Replica: 98ea7859ace84198923d82a57e32849b, State: Running, Role: LEADER
I20260810 22:58:57.170182 12471 consensus_queue.cc:237] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } }
I20260810 22:58:57.173106 11912 catalog_manager.cc:5495] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b reported cstate change: term changed from 0 to 1, leader changed from <none> to 98ea7859ace84198923d82a57e32849b (127.29.78.131). New cstate: current_term: 1 leader_uuid: "98ea7859ace84198923d82a57e32849b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98ea7859ace84198923d82a57e32849b" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 } health_report { overall_health: UNKNOWN } } }
I20260810 22:58:57.202659 12462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:57.199115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.29.78.131:41517)
0810 22:58:57.199282 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:57.199289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:57.199305 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:58:57.199388 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:57.199391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:57.199440 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:57.199545 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:58:57.199785 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.200262 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.201092 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:57.201299 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:57.201405 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:57.201718 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:58:57.202260 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:58:57.202394 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:57.202466 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 22:58:57.202523 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 22:58:57.210110 12087 raft_consensus.cc:3037] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:58:57.211755 12087 raft_consensus.cc:1261] T b3a4f3c2bbb74449afc2a7dc2728b072 P fb1ce61bf4704eb496d6b178061c6a9b [term 1 FOLLOWER]: Refusing update from remote peer 98ea7859ace84198923d82a57e32849b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:58:57.212556 12471 consensus_queue.cc:1035] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1ce61bf4704eb496d6b178061c6a9b" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:58:57.220446 12239 raft_consensus.cc:1261] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 1 FOLLOWER]: Refusing update from remote peer 98ea7859ace84198923d82a57e32849b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:58:57.221268 12471 consensus_queue.cc:1035] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12be0fcf1c1943249632f75e9d9b89ab" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:58:57.228394 12487 mvcc.cc:205] Tried to move back new op lower bound from 7317105611603701760 to 7317105611451461632. Current Snapshot: MvccSnapshot[applied={T|T < 7317105611603701760 or (T in {7317105611603701760})}]
I20260810 22:58:57.228977 12482 mvcc.cc:205] Tried to move back new op lower bound from 7317105611603701760 to 7317105611451461632. Current Snapshot: MvccSnapshot[applied={T|T < 7317105611603701760}]
I20260810 22:58:57.240392 11912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:58:57.240581 11912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:58:57.242949 11912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34ffc03f9cd84a0ca6f6f925e8efcbfd: Sending DeleteTablet for 3 replicas of tablet b3a4f3c2bbb74449afc2a7dc2728b072
I20260810 22:58:57.243779 12200 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a4f3c2bbb74449afc2a7dc2728b072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:58:57 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:55972
I20260810 22:58:57.243769 12360 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a4f3c2bbb74449afc2a7dc2728b072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:58:57 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:52370
I20260810 22:58:57.243783 12066 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a4f3c2bbb74449afc2a7dc2728b072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:58:57 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:52634
I20260810 22:58:57.244616 12505 tablet_replica.cc:330] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: stopping tablet replica
I20260810 22:58:57.244977 12505 raft_consensus.cc:2229] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:58:57.245203 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 11965
I20260810 22:58:57.246641 12505 raft_consensus.cc:2258] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:58:57.248585 12505 ts_tablet_manager.cc:1905] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:58:57.251583 12505 ts_tablet_manager.cc:1918] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:58:57.251744 12505 log.cc:1196] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/b3a4f3c2bbb74449afc2a7dc2728b072
I20260810 22:58:57.252069 12505 ts_tablet_manager.cc:1939] T b3a4f3c2bbb74449afc2a7dc2728b072 P 12be0fcf1c1943249632f75e9d9b89ab: Deleting consensus metadata
I20260810 22:58:57.253208 11903 catalog_manager.cc:4841] TS 12be0fcf1c1943249632f75e9d9b89ab (127.29.78.130:43163): tablet b3a4f3c2bbb74449afc2a7dc2728b072 (table test-table [id=690a721e1e5f464687217d9ddb69df1c]) successfully deleted
I20260810 22:58:57.256976 12502 tablet_replica.cc:330] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b: stopping tablet replica
I20260810 22:58:57.257416 12502 raft_consensus.cc:2229] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:58:57.257936 12502 raft_consensus.cc:2258] T b3a4f3c2bbb74449afc2a7dc2728b072 P 98ea7859ace84198923d82a57e32849b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 22:58:57.258364 11904 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:40685 (error 108)
I20260810 22:58:57.259205 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 12141
I20260810 22:58:57.262857 12461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:57.262471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40685 (local address 127.0.0.1:52648)
0810 22:58:57.262676 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:57.262753 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260810 22:58:57.263152 11904 catalog_manager.cc:4584] TS fb1ce61bf4704eb496d6b178061c6a9b (127.29.78.129:40685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b3a4f3c2bbb74449afc2a7dc2728b072: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40685: connect: Connection refused (error 111)
I20260810 22:58:57.267417 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 12290
I20260810 22:58:57.275836 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 11884
I20260810 22:58:57.286942 30010 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 22:58:57.632354 30010 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 22:58:57.632478 30010 mini_ranger.cc:66] Stopping Ranger...
I20260810 22:58:57.978080 30010 mini_ranger.cc:68] Stopped Ranger
2026-08-10 22:58:57.978 UTC [10829] LOG: received smart shutdown request
2026-08-10 22:58:57.982 UTC [10829] LOG: background worker "logical replication launcher" (PID 10839) exited with exit code 1
2026-08-10 22:58:57.983 UTC [10833] LOG: shutting down
2026-08-10 22:58:58.156 UTC [10829] LOG: database system is shut down
2026-08-10T22:58:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48667 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:58:58 dist-test-slave-6nmv krb5kdc[12516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:58:58 dist-test-slave-6nmv krb5kdc[12516](info): set up 2 sockets
Aug 10 22:58:58 dist-test-slave-6nmv krb5kdc[12516](info): commencing operation
krb5kdc: starting...
W20260810 22:58:59.351948 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:58:59 dist-test-slave-6nmv krb5kdc[12516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:58:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:58:59.472932 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45433
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45433,127.29.78.189:39341,127.29.78.188:46171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:58:59.578368 12532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:59.578652 12532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:59.578724 12532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:59.582569 12532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:58:59.582647 12532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:59.582684 12532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:59.582758 12532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:58:59.582795 12532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:58:59.587785 12532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45433,127.29.78.189:39341,127.29.78.188:46171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:59.588837 12532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:59.589922 12532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:59.598371 12542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:59.598417 12538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:59.598417 12540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:59.598454 12532 server_base.cc:1034] running on GCE node
I20260810 22:58:59.599180 12532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:59.600322 12532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:59.601519 12532 hybrid_clock.cc:648] HybridClock initialized: now 1786402739601502 us; error 30 us; skew 500 ppm
Aug 10 22:58:59 dist-test-slave-6nmv krb5kdc[12516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:59.604503 12532 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:58:59.605338 12532 webserver.cc:460] Webserver started at http://127.29.78.190:34495/ using document root <none> and password file <none>
I20260810 22:58:59.605950 12532 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:59.606046 12532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:59.606287 12532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:58:59.608388 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "ce0d192751cc4bfcbe7c1e7d6379289e"
format_stamp: "Formatted at 2026-08-10 22:58:59 on dist-test-slave-6nmv"
server_key: "4710d75c09344dcaba6e6754957cfa55"
server_key_iv: "1e85a6b41ee9c69ffd047cd573d9f7c8"
server_key_version: "encryptionkey@0"
I20260810 22:58:59.608947 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "ce0d192751cc4bfcbe7c1e7d6379289e"
format_stamp: "Formatted at 2026-08-10 22:58:59 on dist-test-slave-6nmv"
server_key: "4710d75c09344dcaba6e6754957cfa55"
server_key_iv: "1e85a6b41ee9c69ffd047cd573d9f7c8"
server_key_version: "encryptionkey@0"
I20260810 22:58:59.612612 12532 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:58:59.614996 12550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:59.616183 12532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:58:59.616323 12532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "ce0d192751cc4bfcbe7c1e7d6379289e"
format_stamp: "Formatted at 2026-08-10 22:58:59 on dist-test-slave-6nmv"
server_key: "4710d75c09344dcaba6e6754957cfa55"
server_key_iv: "1e85a6b41ee9c69ffd047cd573d9f7c8"
server_key_version: "encryptionkey@0"
I20260810 22:58:59.616431 12532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:59.634282 12532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:59.637290 12532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:59.637495 12532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:59.644073 12532 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45433
I20260810 22:58:59.644140 12608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45433 every 8 connection(s)
I20260810 22:58:59.644912 12532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:58:59.647512 12609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:59.648700 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 12532
I20260810 22:58:59.648847 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:58:59.649154 30010 external_mini_cluster.cc:1428] Setting key 6d3afd76231e67e090444d7ebf56d07f
I20260810 22:58:59.651446 12609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.78.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.189@KRBTEST.COM" created.
Aug 10 22:58:59 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:59.666853 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.654050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.29.78.190:45433)
0810 22:58:59.654552 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:59.654568 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:59.654786 (+ 218us) server_negotiation.cc:407] Connection header received
0810 22:58:59.656138 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:59.656142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:59.656289 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:59.656422 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:59.657486 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.658455 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.659481 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.659790 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.659892 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.662265 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:59.662272 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:59.662285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:59.662313 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:59.663898 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.664520 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.664529 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.664533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.664672 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.665041 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.665060 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.665061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.665370 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:59.665467 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:59.665969 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 22:58:59.666171 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":384,"spinlock_wait_cycles":39808,"thread_start_us":135,"threads_started":1}
I20260810 22:58:59.666855 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.654050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45433 (local address 127.0.0.1:46616)
0810 22:58:59.654574 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.654631 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:59.655843 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:59.656511 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:59.656516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:59.656962 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:59.657324 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.657347 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.658581 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.658584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.659363 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.659384 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.659931 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.659934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.660014 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.660460 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:59.660503 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:59.662111 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:59.664011 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.664023 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.664035 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.664381 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.664773 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.664777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.664781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.664951 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.665492 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:59.665501 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:59.665671 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 22:58:59.665928 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 22:58:59.666260 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":47,"thread_start_us":135,"threads_started":1}
I20260810 22:58:59.669063 12609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:58:59.669770 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.669562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34436)
0810 22:58:59.669661 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.669722 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 22:58:59.670043 12553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
I20260810 22:58:59.670823 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.670621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34442)
0810 22:58:59.670735 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.670777 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 22:58:59.671027 12609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39341: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
I20260810 22:58:59.676478 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:39341
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45433,127.29.78.189:39341,127.29.78.188:46171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:58:59.720286 12609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } attempt: 1
I20260810 22:58:59.721256 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.720878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34452)
0810 22:58:59.721100 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.721162 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 22:58:59.722052 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.721849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34466)
0810 22:58:59.721951 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.721996 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 22:58:59.722224 12609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39341: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
W20260810 22:58:59.785449 12616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:59.785743 12616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:59.785835 12616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:58:59.789712 12616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:58:59.789801 12616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:58:59.789857 12616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:58:59.789930 12616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:58:59.789994 12616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:58:59.790406 12609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } attempt: 2
I20260810 22:58:59.791407 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.791009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34482)
0810 22:58:59.791219 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.791298 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 22:58:59.792230 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.791990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34486)
0810 22:58:59.792104 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.792158 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 22:58:59.792372 12609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39341: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39341: connect: Connection refused (error 111)
I20260810 22:58:59.796205 12616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45433,127.29.78.189:39341,127.29.78.188:46171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:39341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:58:59.797150 12616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:58:59.798171 12616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:58:59.806094 12621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:59.806330 12616 server_base.cc:1034] running on GCE node
W20260810 22:58:59.806176 12624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:58:59.806196 12622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:58:59.807080 12616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:58:59.808120 12616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:58:59.809280 12616 hybrid_clock.cc:648] HybridClock initialized: now 1786402739809253 us; error 36 us; skew 500 ppm
Aug 10 22:58:59 dist-test-slave-6nmv krb5kdc[12516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:58:59.812143 12616 init.cc:377] Logged in from keytab as kudu/127.29.78.189@KRBTEST.COM (short username kudu)
I20260810 22:58:59.812969 12616 webserver.cc:460] Webserver started at http://127.29.78.189:34179/ using document root <none> and password file <none>
I20260810 22:58:59.813481 12616 fs_manager.cc:359] Metadata directory not provided
I20260810 22:58:59.813570 12616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:58:59.813881 12616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:58:59.815776 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data/instance:
uuid: "07826a4826c44c708d6bbb9d75263a0b"
format_stamp: "Formatted at 2026-08-10 22:58:59 on dist-test-slave-6nmv"
server_key: "dcf8b95b3eb0795ae02f6d37945ce2a0"
server_key_iv: "b704ba36eb85524e208ae504012a11ce"
server_key_version: "encryptionkey@0"
I20260810 22:58:59.816326 12616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/wal/instance:
uuid: "07826a4826c44c708d6bbb9d75263a0b"
format_stamp: "Formatted at 2026-08-10 22:58:59 on dist-test-slave-6nmv"
server_key: "dcf8b95b3eb0795ae02f6d37945ce2a0"
server_key_iv: "b704ba36eb85524e208ae504012a11ce"
server_key_version: "encryptionkey@0"
I20260810 22:58:59.819995 12616 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:58:59.822587 12632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:58:59.823681 12616 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:58:59.823817 12616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/wal
uuid: "07826a4826c44c708d6bbb9d75263a0b"
format_stamp: "Formatted at 2026-08-10 22:58:59 on dist-test-slave-6nmv"
server_key: "dcf8b95b3eb0795ae02f6d37945ce2a0"
server_key_iv: "b704ba36eb85524e208ae504012a11ce"
server_key_version: "encryptionkey@0"
I20260810 22:58:59.823912 12616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:58:59.836410 12616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:58:59.839488 12616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:58:59.839675 12616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:58:59.846277 12616 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:39341
I20260810 22:58:59.846292 12689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:39341 every 8 connection(s)
I20260810 22:58:59.847226 12616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
I20260810 22:58:59.849926 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:58:59.852299 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 12616
I20260810 22:58:59.852442 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-1/wal/instance
I20260810 22:58:59.852761 30010 external_mini_cluster.cc:1428] Setting key f6d29371149a5370ca05471dbe76c88a
I20260810 22:58:59.854061 12690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.78.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.188@KRBTEST.COM" created.
Aug 10 22:58:59 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:58:59.870455 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.856216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.29.78.190:45433)
0810 22:58:59.856346 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:59.856349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:59.857210 (+ 861us) server_negotiation.cc:407] Connection header received
0810 22:58:59.858234 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:59.858238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:59.858313 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:59.858420 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:59.859534 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.860033 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.861150 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.861394 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.861514 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.866273 (+ 4759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:59.866277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:59.866279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:59.866302 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:59.868151 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.868819 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.868822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.868823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.868902 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.869316 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.869317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.869318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.869484 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:59.869555 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:59.870243 (+ 688us) server_negotiation.cc:299] Negotiation successful
0810 22:58:59.870334 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 22:58:59.871639 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.856564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45433 (local address 127.0.0.1:46622)
0810 22:58:59.857026 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.857083 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:59.858010 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:59.858583 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:59.858590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:59.858931 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:59.859390 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.859402 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.860168 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.860177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.861015 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.861025 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.861567 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.861571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.861790 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.862525 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:59.862543 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:59.866089 (+ 3546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:59.868290 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.868295 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.868330 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.868705 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.869023 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.869028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.869033 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.869222 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.869592 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:59.869601 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:59.869949 (+ 348us) client_negotiation.cc:769] Sending connection context
0810 22:58:59.870201 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 22:58:59.870518 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":134,"threads_started":1}
I20260810 22:58:59.873057 12690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:58:59 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
I20260810 22:58:59.882296 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.873662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.29.78.189:39341)
0810 22:58:59.874023 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:59.874032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:59.874048 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:58:59.874077 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:59.874080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:59.874210 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:59.874345 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:59.874680 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.875442 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.876185 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.876573 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.876690 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.878706 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:59.878714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:59.878725 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:59.878747 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:59.880340 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.880819 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.880823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.880824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.880905 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.881167 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.881169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.881170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.881467 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:59.881585 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:59.881950 (+ 365us) server_negotiation.cc:299] Negotiation successful
0810 22:58:59.882126 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":257,"thread_start_us":140,"threads_started":1}
I20260810 22:58:59.882304 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.873551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34490)
0810 22:58:59.873722 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.873733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:59.873866 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:59.874324 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:59.874327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:59.874498 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:59.874566 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.874571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.875556 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.875562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.876071 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.876078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.876688 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.876691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.876836 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.877230 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:59.877241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:59.878572 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:59.880448 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.880451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.880452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.880688 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.880998 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.881000 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.881001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.881077 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.881574 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:59.881576 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:59.881656 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 22:58:59.881792 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 22:58:59.882006 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":40}
I20260810 22:58:59.883395 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:46171
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45433,127.29.78.189:39341,127.29.78.188:46171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:58:59.883533 12609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } attempt: 3
I20260810 22:58:59.884497 12690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:58:59.885317 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.885025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49304)
0810 22:58:59.885137 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.885235 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260810 22:58:59.885485 12636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
I20260810 22:58:59.886734 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.886153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49314)
0810 22:58:59.886379 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.886428 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":90,"threads_started":1}
W20260810 22:58:59.887099 12690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:46171: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Aug 10 22:58:59 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260810 22:58:59.893914 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.884132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.29.78.189:39341)
0810 22:58:59.884227 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 22:58:59.884230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:58:59.884245 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:58:59.884402 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:58:59.884407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:58:59.884455 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:58:59.885058 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:58:59.885156 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.885858 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.887808 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.888223 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.888347 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.890725 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:58:59.890728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:58:59.890730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:58:59.890750 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:58:59.892198 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.892725 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.892728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.892729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.892810 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:58:59.893120 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:58:59.893122 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:58:59.893123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:58:59.893262 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:58:59.893329 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 22:58:59.893653 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 22:58:59.893761 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 22:58:59.893987 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.883989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34498)
0810 22:58:59.884189 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.884205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:58:59.884281 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:58:59.884576 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:58:59.884581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:58:59.884773 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:58:59.884932 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.884942 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.886029 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.886034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.887647 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:58:59.887659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:58:59.888350 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:58:59.888353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:58:59.888439 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:58:59.888928 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:58:59.888940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:58:59.890592 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:58:59.892315 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.892321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.892326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.892627 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.892913 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:58:59.892917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:58:59.892919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:58:59.893023 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:58:59.893354 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:58:59.893359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:58:59.893484 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 22:58:59.893615 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 22:58:59.893776 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 22:58:59.894870 12609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
I20260810 22:58:59.895633 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.895393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49316)
0810 22:58:59.895529 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.895576 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 22:58:59.896440 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.896196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49332)
0810 22:58:59.896348 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.896390 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260810 22:58:59.896643 12609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:46171: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
I20260810 22:58:59.927820 12609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } attempt: 1
I20260810 22:58:59.928737 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.928447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49348)
0810 22:58:59.928598 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.928659 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 22:58:59.929457 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.929244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49356)
0810 22:58:59.929350 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.929402 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 22:58:59.929603 12609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:46171: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
I20260810 22:58:59.936367 12690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } attempt: 1
I20260810 22:58:59.937091 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.936817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49368)
0810 22:58:59.936946 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.937021 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 22:58:59.937904 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:58:59.937631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49372)
0810 22:58:59.937734 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 22:58:59.937778 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 22:58:59.938043 12690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:46171: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
W20260810 22:58:59.996608 12697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:58:59.996824 12697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:58:59.996894 12697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:00.000737 12697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:59:00.000804 12697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:00.000823 12697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:00.000880 12697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:59:00.000914 12697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:59:00.004799 12609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } attempt: 2
I20260810 22:59:00.005741 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.005399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49382)
0810 22:59:00.005595 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.005660 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 22:59:00.006220 12690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } attempt: 2
I20260810 22:59:00.006618 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.006321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49398)
0810 22:59:00.006511 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.006559 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 22:59:00.006906 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.006606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49404)
0810 22:59:00.006770 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.006837 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 22:59:00.007072 12609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:46171: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
I20260810 22:59:00.007380 12697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45433,127.29.78.189:39341,127.29.78.188:46171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:46171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:00.007710 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.007426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49416)
0810 22:59:00.007571 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.007638 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 22:59:00.007865 12690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:46171: Network error: Client connection negotiation failed: client connection to 127.29.78.188:46171: connect: Connection refused (error 111)
I20260810 22:59:00.008591 12697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:00.009649 12697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:00.018496 12705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:00.018507 12707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:00.018585 12697 server_base.cc:1034] running on GCE node
W20260810 22:59:00.018515 12704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:00.019239 12697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:00.020401 12697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:00.021570 12697 hybrid_clock.cc:648] HybridClock initialized: now 1786402740021551 us; error 30 us; skew 500 ppm
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402740, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:00.024545 12697 init.cc:377] Logged in from keytab as kudu/127.29.78.188@KRBTEST.COM (short username kudu)
I20260810 22:59:00.025477 12697 webserver.cc:460] Webserver started at http://127.29.78.188:45543/ using document root <none> and password file <none>
I20260810 22:59:00.026118 12697 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:00.026216 12697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:00.026568 12697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:00.028735 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data/instance:
uuid: "2f0b198efc054e0b8f9464a08e9d1fd3"
format_stamp: "Formatted at 2026-08-10 22:59:00 on dist-test-slave-6nmv"
server_key: "1b284c2ce176aec73b21b4ad937a46e5"
server_key_iv: "662408fc5ebd02ed7a12f486f45825e3"
server_key_version: "encryptionkey@0"
I20260810 22:59:00.029307 12697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/wal/instance:
uuid: "2f0b198efc054e0b8f9464a08e9d1fd3"
format_stamp: "Formatted at 2026-08-10 22:59:00 on dist-test-slave-6nmv"
server_key: "1b284c2ce176aec73b21b4ad937a46e5"
server_key_iv: "662408fc5ebd02ed7a12f486f45825e3"
server_key_version: "encryptionkey@0"
I20260810 22:59:00.033252 12697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 22:59:00.035756 12714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:00.036972 12697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:00.037125 12697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/wal
uuid: "2f0b198efc054e0b8f9464a08e9d1fd3"
format_stamp: "Formatted at 2026-08-10 22:59:00 on dist-test-slave-6nmv"
server_key: "1b284c2ce176aec73b21b4ad937a46e5"
server_key_iv: "662408fc5ebd02ed7a12f486f45825e3"
server_key_version: "encryptionkey@0"
I20260810 22:59:00.037236 12697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:00.054911 12697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:00.058111 12697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:00.058318 12697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:00.065688 12697 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:46171
I20260810 22:59:00.065717 12772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:46171 every 8 connection(s)
I20260810 22:59:00.066697 12697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
I20260810 22:59:00.069442 12773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:00.071998 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 12697
I20260810 22:59:00.072114 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402578782454-30010-0/minicluster-data/master-2/wal/instance
I20260810 22:59:00.072443 30010 external_mini_cluster.cc:1428] Setting key 31026606cb5c84ed110b9e87b9506ccf
I20260810 22:59:00.073536 12773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402740, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260810 22:59:00.086627 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.075433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.29.78.190:45433)
0810 22:59:00.075598 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.075602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.075918 (+ 316us) server_negotiation.cc:407] Connection header received
0810 22:59:00.076157 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.076161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.076212 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.076297 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.077268 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.078062 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.079442 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.079776 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.079887 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.082733 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.082736 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.082738 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.082765 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.084480 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.085068 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.085072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.085074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.085161 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.085513 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.085516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.085518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.085722 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.085860 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.086307 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.086436 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 22:59:00.087488 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.073952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49426 (local address 127.29.78.188:46171)
0810 22:59:00.074397 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.074412 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.074447 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:59:00.075166 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.075192 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.075627 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.075948 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.076909 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.078280 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.079434 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.080002 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.080092 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.083163 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.083168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.083179 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.083199 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.084788 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.085224 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.085230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.085231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.085347 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.085639 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.085644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.085645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.086073 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.086300 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.086508 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.086646 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"spinlock_wait_cycles":768,"thread_start_us":127,"threads_started":1}
I20260810 22:59:00.087487 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.075365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:45433 (local address 127.0.0.1:46624)
0810 22:59:00.075758 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.075802 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:00.076038 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:00.076382 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:00.076390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:00.076859 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:00.077115 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.077126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.078165 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.078168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.079322 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.079336 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.079923 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.079926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.080095 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.080742 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:00.080768 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:00.082577 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:00.084590 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.084599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.084611 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.084957 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.085258 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.085262 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.085265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.085397 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.085854 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:00.085860 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:00.086086 (+ 226us) client_negotiation.cc:769] Sending connection context
0810 22:59:00.086263 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 22:59:00.086567 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"spinlock_wait_cycles":3456,"thread_start_us":153,"threads_started":1}
I20260810 22:59:00.088699 12773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402740, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:00.098070 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.089215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39341 (local address 127.0.0.1:34504)
0810 22:59:00.089492 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.089507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:00.089594 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:00.089935 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:00.089937 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:00.090078 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:00.090215 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.090222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.091275 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.091278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.092037 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.092046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.092603 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.092606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.092684 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.093149 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:00.093163 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:00.094447 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:00.096560 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.096563 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.096564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.096757 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.097050 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.097054 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.097056 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.097143 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.097531 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:00.097535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:00.097632 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 22:59:00.097739 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 22:59:00.097909 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260810 22:59:00.098083 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.089351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.29.78.189:39341)
0810 22:59:00.089494 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.089500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.089552 (+ 52us) server_negotiation.cc:407] Connection header received
0810 22:59:00.089701 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.089707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.089764 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.089915 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.090341 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.091176 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.092150 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.092492 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.092596 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.094598 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.094604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.094610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.094639 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.096435 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.096858 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.096871 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.096873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.096957 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.097235 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.097240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.097242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.097423 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.097530 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.097793 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.097944 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 22:59:00.098997 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.089659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46634 (local address 127.29.78.190:45433)
0810 22:59:00.089756 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.089759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.090144 (+ 385us) server_negotiation.cc:407] Connection header received
0810 22:59:00.090207 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.090210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.090255 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.090336 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.091069 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.091761 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.092754 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.093059 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.093158 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.095681 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.095686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.095690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.095715 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.097230 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.097688 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.097696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.097697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.097769 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.098133 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.098137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.098138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.098355 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.098597 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.098714 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.098832 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 22:59:00.099052 12690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } attempt: 3
I20260810 22:59:00.099054 12773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402740, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260810 22:59:00.106245 12609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } attempt: 3
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260810 22:59:00.108866 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.099513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49446)
0810 22:59:00.099597 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.099613 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:00.099683 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:00.100208 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:00.100211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:00.100386 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:00.100471 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.100478 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.101313 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.101318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.102078 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.102088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.102684 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.102688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.102765 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.103231 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:00.103245 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:00.104523 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:00.107144 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.107147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.107148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.107478 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.107786 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.107791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.107794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.107887 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.108309 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:00.108312 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:00.108392 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 22:59:00.108548 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 22:59:00.108698 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":5}
I20260810 22:59:00.109627 12778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.099644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.29.78.188:46171)
0810 22:59:00.100007 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.100013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.100026 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:59:00.100055 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.100058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.100109 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.100198 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.100591 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.101212 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.102195 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.102558 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.102672 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.104624 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.104627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.104630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.104649 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.107030 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.107583 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.107586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.107588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.107682 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.108008 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.108011 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.108012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.108208 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.109279 (+ 1071us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.109356 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.109475 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"spinlock_wait_cycles":4608,"thread_start_us":125,"threads_started":1}
I20260810 22:59:00.110558 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.099454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49436)
0810 22:59:00.099745 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.099762 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:00.099850 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:00.100241 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:00.100244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:00.100457 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:00.100610 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.100622 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.102201 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.102204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.103021 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.103031 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.103644 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.103648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.103790 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.104263 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:00.104284 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:00.105680 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:00.108800 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.108806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.108808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.109073 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.109426 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.109429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.109431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.109521 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.110029 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:00.110032 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:00.110130 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 22:59:00.110237 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 22:59:00.110391 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":25}
I20260810 22:59:00.111330 12781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.099614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.29.78.188:46171)
0810 22:59:00.099975 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.099987 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.099998 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:59:00.100050 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.100052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.100099 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.100726 (+ 627us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.100819 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.102078 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.103153 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.103504 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.103610 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.105883 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.105890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.105896 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.105928 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.108666 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.109196 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.109202 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.109207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.109309 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.109640 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.109646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.109649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.109902 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.110030 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.111072 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.111184 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"spinlock_wait_cycles":2944}
I20260810 22:59:00.112152 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.100470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34516 (local address 127.29.78.189:39341)
0810 22:59:00.100581 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.100585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.100634 (+ 49us) server_negotiation.cc:407] Connection header received
0810 22:59:00.100754 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.100757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.100795 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.100870 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.101662 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.102412 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.103411 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.104353 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.104462 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.108111 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.108116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.108121 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.108149 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.109790 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.110269 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.110273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.110275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.110422 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.110763 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.110767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.110768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.110991 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.111205 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.111885 (+ 680us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.111993 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":539,"server-negotiator.queue_time_us":44}
Aug 10 22:59:00 dist-test-slave-6nmv krb5kdc[12516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402739, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260810 22:59:00.113991 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: Bootstrap starting.
I20260810 22:59:00.116302 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:59:00.117153 12773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:00.117427 12612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.106708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:46171 (local address 127.0.0.1:49456)
0810 22:59:00.106830 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:00.106847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:00.106931 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:00.107298 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:00.107301 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:00.107454 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:00.107571 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.107578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.109455 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.109459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.110267 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:00.110278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.111609 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.111613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:00.111700 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.112150 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:00.112166 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:00.113594 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:00.115829 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.115832 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.115834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.116046 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.116347 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:00.116351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:00.116353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:00.116453 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:00.116871 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:00.116874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:00.116977 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 22:59:00.117113 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 22:59:00.117260 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 22:59:00.117517 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.106765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49456 (local address 127.29.78.188:46171)
0810 22:59:00.107043 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.107055 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.107070 (+ 15us) server_negotiation.cc:407] Connection header received
0810 22:59:00.107106 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.107111 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.107154 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.107973 (+ 819us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.108067 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.109328 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.111090 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.111433 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.112681 (+ 1248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.113880 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.113884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.113886 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.113911 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.115718 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.116150 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.116154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.116156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.116256 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.116557 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.116560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.116561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.116755 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.116860 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.117186 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.117347 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260810 22:59:00.120223 12609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: Bootstrap starting.
I20260810 22:59:00.120223 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b: Bootstrap starting.
I20260810 22:59:00.123214 12690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:00.123220 12609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:00.124127 12690 log.cc:824] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:00.124127 12609 log.cc:824] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:00.124881 12773 log.cc:824] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:00.126817 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b: No bootstrap required, opened a new log
I20260810 22:59:00.127254 12609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: No bootstrap required, opened a new log
I20260810 22:59:00.127276 12782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.119106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.29.78.188:46171)
0810 22:59:00.119207 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.119211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.119595 (+ 384us) server_negotiation.cc:407] Connection header received
0810 22:59:00.119696 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.119699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.119736 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.119824 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.120538 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.121030 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.121862 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.122189 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.122450 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.123712 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.123715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.123718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.123741 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.125291 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.125792 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.125796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.125799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.125909 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.126241 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.126244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.126246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.126462 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.126571 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.127022 (+ 451us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.127139 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 22:59:00.130795 12690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.131134 12690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:00.131238 12690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07826a4826c44c708d6bbb9d75263a0b, State: Initialized, Role: FOLLOWER
I20260810 22:59:00.132421 12609 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.132682 12609 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:00.132748 12609 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0d192751cc4bfcbe7c1e7d6379289e, State: Initialized, Role: FOLLOWER
I20260810 22:59:00.133020 12773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: No bootstrap required, opened a new log
I20260810 22:59:00.133224 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.118730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.29.78.189:39341)
0810 22:59:00.124605 (+ 5875us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.124610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.124624 (+ 14us) server_negotiation.cc:407] Connection header received
0810 22:59:00.124658 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.124661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.124705 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.125536 (+ 831us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.125627 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.126315 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.127442 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.127733 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.127840 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.129511 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.129515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.129518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.129543 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.131241 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.131756 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.131759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.131761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.131850 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.132175 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.132178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.132180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.132386 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.132503 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.132942 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.133085 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5801}
I20260810 22:59:00.133387 12609 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.134603 12609 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:00.137799 12773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.138526 12690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
W20260810 22:59:00.139019 12556 tablet.cc:2369] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:00.139983 12690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:00.140254 12773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:00.140316 12773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0b198efc054e0b8f9464a08e9d1fd3, State: Initialized, Role: FOLLOWER
W20260810 22:59:00.140393 12804 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:59:00.140460 12804 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:59:00.141872 12773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.142294 12613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:00.131782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.29.78.190:45433)
0810 22:59:00.131886 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:00.131890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:00.132279 (+ 389us) server_negotiation.cc:407] Connection header received
0810 22:59:00.132445 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:00.132448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:00.132496 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:00.132546 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:00.133499 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.134217 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.135838 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:00.136175 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:00.136302 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:00.137935 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:00.137939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:00.137942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:00.137967 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:00.139913 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.140462 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.140465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.140467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.140557 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:00.140887 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:00.140890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:00.140892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:00.141162 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:00.141334 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:00.142028 (+ 694us) server_negotiation.cc:299] Negotiation successful
0810 22:59:00.142144 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:59:00.143205 12773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:00.145476 12807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.145694 12807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: This master's current role is: FOLLOWER
W20260810 22:59:00.145725 12822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:59:00.145797 12822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:59:00.146027 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.146214 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:59:00.139637 12532 master_runner.cc:186] Error getting master registration for 127.29.78.189:39341: OK, instance_id { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" instance_seqno: 1786402739844418 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 22:59:00.147326 12532 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 22:59:00.148854 12834 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:59:00.148931 12834 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:59:00.149391 12792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.149647 12616 master_runner.cc:263] No leader master found from master 07826a4826c44c708d6bbb9d75263a0b
I20260810 22:59:00.149705 12616 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:59:00.151134 12697 master_runner.cc:263] No leader master found from master 2f0b198efc054e0b8f9464a08e9d1fd3
I20260810 22:59:00.151196 12697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 22:59:00.151294 12792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:59:00.177706 12807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:00.177947 12807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.179026 12807 leader_election.cc:288] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce0d192751cc4bfcbe7c1e7d6379289e (127.29.78.190:45433), 2f0b198efc054e0b8f9464a08e9d1fd3 (127.29.78.188:46171)
I20260810 22:59:00.179894 12579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07826a4826c44c708d6bbb9d75263a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" is_pre_election: true
I20260810 22:59:00.180210 12579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07826a4826c44c708d6bbb9d75263a0b in term 0.
I20260810 22:59:00.180786 12635 leader_election.cc:302] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07826a4826c44c708d6bbb9d75263a0b, ce0d192751cc4bfcbe7c1e7d6379289e; no voters:
I20260810 22:59:00.181465 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07826a4826c44c708d6bbb9d75263a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" is_pre_election: true
I20260810 22:59:00.181728 12746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07826a4826c44c708d6bbb9d75263a0b in term 0.
I20260810 22:59:00.182389 12807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:59:00.182469 12807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:59:00.182510 12807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:00.183880 12807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.184816 12746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07826a4826c44c708d6bbb9d75263a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3"
I20260810 22:59:00.184921 12746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:00.185959 12579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "07826a4826c44c708d6bbb9d75263a0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e"
I20260810 22:59:00.186090 12579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:00.187868 12579 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07826a4826c44c708d6bbb9d75263a0b in term 1.
I20260810 22:59:00.188254 12635 leader_election.cc:302] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07826a4826c44c708d6bbb9d75263a0b, ce0d192751cc4bfcbe7c1e7d6379289e; no voters:
I20260810 22:59:00.188923 12746 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07826a4826c44c708d6bbb9d75263a0b in term 1.
I20260810 22:59:00.191787 12807 leader_election.cc:288] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [CANDIDATE]: Term 1 election: Requested vote from peers ce0d192751cc4bfcbe7c1e7d6379289e (127.29.78.190:45433), 2f0b198efc054e0b8f9464a08e9d1fd3 (127.29.78.188:46171)
I20260810 22:59:00.192027 12807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:00.192157 12807 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [term 1 LEADER]: Becoming Leader. State: Replica: 07826a4826c44c708d6bbb9d75263a0b, State: Running, Role: LEADER
I20260810 22:59:00.192541 12807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } }
I20260810 22:59:00.194530 12807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07826a4826c44c708d6bbb9d75263a0b. Latest consensus state: current_term: 1 leader_uuid: "07826a4826c44c708d6bbb9d75263a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.194698 12807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:00.195225 12839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:59:00.196225 12839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:59:00.206485 12746 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [term 1 FOLLOWER]: Refusing update from remote peer 07826a4826c44c708d6bbb9d75263a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:00.207219 12837 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:00.210968 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07826a4826c44c708d6bbb9d75263a0b. Latest consensus state: current_term: 1 leader_uuid: "07826a4826c44c708d6bbb9d75263a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.211078 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:59:00.212154 12579 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [term 1 FOLLOWER]: Refusing update from remote peer 07826a4826c44c708d6bbb9d75263a0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:00.212792 12837 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:00.219036 12792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07826a4826c44c708d6bbb9d75263a0b. Latest consensus state: current_term: 1 leader_uuid: "07826a4826c44c708d6bbb9d75263a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.219172 12792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:59:00.225580 12837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07826a4826c44c708d6bbb9d75263a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.225708 12837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:00.227156 12792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07826a4826c44c708d6bbb9d75263a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.227283 12792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:59:00.227735 12841 mvcc.cc:205] Tried to move back new op lower bound from 7317105623864610816 to 7317105623832203264. Current Snapshot: MvccSnapshot[applied={T|T < 7317105623864610816}]
I20260810 22:59:00.228188 12844 mvcc.cc:205] Tried to move back new op lower bound from 7317105623864610816 to 7317105623832203264. Current Snapshot: MvccSnapshot[applied={T|T < 7317105623864610816}]
I20260810 22:59:00.230700 12807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "07826a4826c44c708d6bbb9d75263a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.230805 12807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:00.232023 12816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "07826a4826c44c708d6bbb9d75263a0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce0d192751cc4bfcbe7c1e7d6379289e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45433 } } peers { permanent_uuid: "07826a4826c44c708d6bbb9d75263a0b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39341 } } peers { permanent_uuid: "2f0b198efc054e0b8f9464a08e9d1fd3" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 46171 } } }
I20260810 22:59:00.232139 12816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 22:59:00.237144 12842 mvcc.cc:205] Tried to move back new op lower bound from 7317105623864610816 to 7317105623832203264. Current Snapshot: MvccSnapshot[applied={T|T < 7317105623864610816}]
I20260810 22:59:00.240104 12839 catalog_manager.cc:1347] Generated new cluster ID: 565dc3473e124375a5d653b097cbe828
I20260810 22:59:00.240164 12839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:59:00.285983 12839 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:59:00.287431 12839 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:59:00.316994 12839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07826a4826c44c708d6bbb9d75263a0b: Generated new TSK 0
I20260810 22:59:00.318034 12839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:59:01.142004 12804 catalog_manager.cc:1259] Loaded cluster ID: 565dc3473e124375a5d653b097cbe828
I20260810 22:59:01.142107 12804 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: loading cluster ID for follower catalog manager: success
I20260810 22:59:01.143769 12804 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: acquiring CA information for follower catalog manager: success
I20260810 22:59:01.144943 12804 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce0d192751cc4bfcbe7c1e7d6379289e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:59:01.150121 12834 catalog_manager.cc:1259] Loaded cluster ID: 565dc3473e124375a5d653b097cbe828
I20260810 22:59:01.150215 12834 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: loading cluster ID for follower catalog manager: success
I20260810 22:59:01.151700 12834 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: acquiring CA information for follower catalog manager: success
I20260810 22:59:01.152745 12834 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f0b198efc054e0b8f9464a08e9d1fd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 22:59:01.386143 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 12532
I20260810 22:59:01.395247 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 12616
I20260810 22:59:01.402981 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 12697
2026-08-10T22:59:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3123 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:59:01 dist-test-slave-6nmv krb5kdc[12854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:59:01 dist-test-slave-6nmv krb5kdc[12854](info): set up 2 sockets
Aug 10 22:59:01 dist-test-slave-6nmv krb5kdc[12854](info): commencing operation
krb5kdc: starting...
W20260810 22:59:02.458988 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:59:02 dist-test-slave-6nmv krb5kdc[12854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:59:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:59:02.580250 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34049
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:02.685674 12867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:02.685989 12867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:02.686069 12867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:02.689996 12867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:59:02.690038 12867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:02.690056 12867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:02.690071 12867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:59:02.690085 12867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:59:02.695092 12867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:02.696055 12867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:02.697086 12867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:02.705217 12875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:02.705225 12874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:02.705269 12867 server_base.cc:1034] running on GCE node
W20260810 22:59:02.705225 12877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:02.706051 12867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:02.707015 12867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:02.708215 12867 hybrid_clock.cc:648] HybridClock initialized: now 1786402742708191 us; error 29 us; skew 500 ppm
Aug 10 22:59:02 dist-test-slave-6nmv krb5kdc[12854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402742, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:02.710965 12867 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:59:02.711807 12867 webserver.cc:460] Webserver started at http://127.29.78.190:43877/ using document root <none> and password file <none>
I20260810 22:59:02.712338 12867 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:02.712446 12867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:02.712663 12867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:02.714635 12867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "0cc4f49095844e26b3e7758cfb97e3da"
format_stamp: "Formatted at 2026-08-10 22:59:02 on dist-test-slave-6nmv"
server_key: "b82c1eedb6b95e41b1297ef7efa8cc0f"
server_key_iv: "070d6ddd850bb7a96dd586737ba18f5e"
server_key_version: "encryptionkey@0"
I20260810 22:59:02.715179 12867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "0cc4f49095844e26b3e7758cfb97e3da"
format_stamp: "Formatted at 2026-08-10 22:59:02 on dist-test-slave-6nmv"
server_key: "b82c1eedb6b95e41b1297ef7efa8cc0f"
server_key_iv: "070d6ddd850bb7a96dd586737ba18f5e"
server_key_version: "encryptionkey@0"
I20260810 22:59:02.718762 12867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:59:02.721086 12884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:02.722241 12867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:02.722370 12867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "0cc4f49095844e26b3e7758cfb97e3da"
format_stamp: "Formatted at 2026-08-10 22:59:02 on dist-test-slave-6nmv"
server_key: "b82c1eedb6b95e41b1297ef7efa8cc0f"
server_key_iv: "070d6ddd850bb7a96dd586737ba18f5e"
server_key_version: "encryptionkey@0"
I20260810 22:59:02.722488 12867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:02.747323 12867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:02.750588 12867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:02.750783 12867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:02.757386 12867 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:34049
I20260810 22:59:02.757445 12940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:34049 every 8 connection(s)
I20260810 22:59:02.758423 12867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:59:02.760962 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:02.765992 12941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da: Bootstrap starting.
I20260810 22:59:02.766418 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 12867
I20260810 22:59:02.766563 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:59:02.766849 30010 external_mini_cluster.cc:1428] Setting key 920634c79c93746b9b0354ddc582e625
I20260810 22:59:02.768525 12941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:02.769237 12941 log.cc:824] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:02.771286 12941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da: No bootstrap required, opened a new log
Aug 10 22:59:02 dist-test-slave-6nmv krb5kdc[12854](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:02.775871 12941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4f49095844e26b3e7758cfb97e3da" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34049 } }
I20260810 22:59:02.776120 12941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:02.776240 12941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc4f49095844e26b3e7758cfb97e3da, State: Initialized, Role: FOLLOWER
I20260810 22:59:02.776731 12941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4f49095844e26b3e7758cfb97e3da" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34049 } }
I20260810 22:59:02.776860 12941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:59:02.776949 12941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:59:02.777069 12941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:02.778033 12941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4f49095844e26b3e7758cfb97e3da" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34049 } }
I20260810 22:59:02.778419 12941 leader_election.cc:302] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cc4f49095844e26b3e7758cfb97e3da; no voters:
I20260810 22:59:02.778820 12941 leader_election.cc:288] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:59:02.778880 12946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:02.779121 12946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [term 1 LEADER]: Becoming Leader. State: Replica: 0cc4f49095844e26b3e7758cfb97e3da, State: Running, Role: LEADER
I20260810 22:59:02.779438 12946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4f49095844e26b3e7758cfb97e3da" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34049 } }
I20260810 22:59:02.780186 12941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:02.780668 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cc4f49095844e26b3e7758cfb97e3da. Latest consensus state: current_term: 1 leader_uuid: "0cc4f49095844e26b3e7758cfb97e3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4f49095844e26b3e7758cfb97e3da" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34049 } } }
I20260810 22:59:02.780786 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:02.781025 12946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cc4f49095844e26b3e7758cfb97e3da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cc4f49095844e26b3e7758cfb97e3da" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 34049 } } }
I20260810 22:59:02.781150 12946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:02.781263 12950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:59:02.781795 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:02.767946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.29.78.190:34049)
0810 22:59:02.768366 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:02.768381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:02.768415 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:59:02.769084 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:02.769100 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:02.769476 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:02.769791 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:02.770619 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:02.771603 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:02.772712 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:02.773133 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:02.773240 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:02.776223 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:02.776228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:02.776241 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:02.776267 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:02.778417 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:02.778891 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:02.778897 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:02.778905 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:02.779055 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:02.779376 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:02.779382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:02.779385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:02.779830 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:02.780079 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:02.780348 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 22:59:02.780703 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":143,"threads_started":1}
I20260810 22:59:02.784432 12950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:59:02.784806 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 22:59:02.790059 12950 catalog_manager.cc:1347] Generated new cluster ID: dfa7a901f53d4053be95260f5c23a153
I20260810 22:59:02.790125 12950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:59:02.801344 12950 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:59:02.802197 12950 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:59:02.812275 12950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cc4f49095844e26b3e7758cfb97e3da: Generated new TSK 0
I20260810 22:59:02.812841 12950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 22:59:02.840732 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 12867
2026-08-10T22:59:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1437 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:59:02 dist-test-slave-6nmv krb5kdc[12971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:59:02 dist-test-slave-6nmv krb5kdc[12971](info): set up 2 sockets
Aug 10 22:59:02 dist-test-slave-6nmv krb5kdc[12971](info): commencing operation
krb5kdc: starting...
W20260810 22:59:03.898335 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:59:03 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:59:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:59:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:59:04.022301 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38533
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:04.127044 12984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:04.127308 12984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:04.127369 12984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:04.131084 12984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:59:04.131168 12984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:04.131204 12984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:04.131256 12984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:59:04.131304 12984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:59:04.136305 12984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:04.137234 12984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:04.138161 12984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:04.146028 12989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:04.146147 12995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:04.146138 12992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:04.146075 12984 server_base.cc:1034] running on GCE node
I20260810 22:59:04.146782 12984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:04.147804 12984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:04.148988 12984 hybrid_clock.cc:648] HybridClock initialized: now 1786402744148971 us; error 27 us; skew 500 ppm
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402744, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:04.155529 12984 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:59:04.156296 12984 webserver.cc:460] Webserver started at http://127.29.78.190:45669/ using document root <none> and password file <none>
I20260810 22:59:04.156823 12984 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:04.156891 12984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:04.157037 12984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:04.158987 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "ea9356702dbe4b21ba44b10a94f8dfb8"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cc5ab17e7955180780ae4100713d4db9"
server_key_iv: "8a544753c146507babdf8413611c1266"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.159502 12984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "ea9356702dbe4b21ba44b10a94f8dfb8"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cc5ab17e7955180780ae4100713d4db9"
server_key_iv: "8a544753c146507babdf8413611c1266"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.163007 12984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 22:59:04.165128 13002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.166363 12984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:04.166497 12984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "ea9356702dbe4b21ba44b10a94f8dfb8"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cc5ab17e7955180780ae4100713d4db9"
server_key_iv: "8a544753c146507babdf8413611c1266"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.166594 12984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:04.181160 12984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:04.183743 12984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:04.183917 12984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:04.190479 12984 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:38533
I20260810 22:59:04.190727 13057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:38533 every 8 connection(s)
I20260810 22:59:04.191331 12984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:59:04.194011 13058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:04.198612 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 12984
I20260810 22:59:04.198776 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:59:04.199025 13058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Bootstrap starting.
I20260810 22:59:04.199056 30010 external_mini_cluster.cc:1428] Setting key e6709b54537f322daa846b2a5b176793
I20260810 22:59:04.201432 13058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:04.202315 13058 log.cc:824] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:04.204281 13058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: No bootstrap required, opened a new log
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:04.208627 13058 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:04.208906 13058 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:04.209002 13058 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9356702dbe4b21ba44b10a94f8dfb8, State: Initialized, Role: FOLLOWER
I20260810 22:59:04.209544 13058 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:04.209705 13058 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:59:04.209774 13058 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:59:04.209890 13058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:04.210706 13058 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:04.211041 13058 leader_election.cc:302] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9356702dbe4b21ba44b10a94f8dfb8; no voters:
I20260810 22:59:04.211382 13058 leader_election.cc:288] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:59:04.211439 13065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:04.211652 13065 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 LEADER]: Becoming Leader. State: Replica: ea9356702dbe4b21ba44b10a94f8dfb8, State: Running, Role: LEADER
I20260810 22:59:04.212003 13065 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:04.212658 13058 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:04.213671 13067 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } } }
I20260810 22:59:04.213788 13067 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:04.214031 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.200279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50182 (local address 127.29.78.190:38533)
0810 22:59:04.200704 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.200718 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.200751 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:59:04.201310 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.201324 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.201660 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.201975 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:04.202883 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.203891 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.205021 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.205447 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.205572 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.208305 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:04.208310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:04.208322 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:04.208347 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:04.210295 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.210735 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.210741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.210746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.210904 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.211206 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.211209 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.211211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.211780 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:04.212543 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.212660 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.212974 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":144,"threads_started":1}
I20260810 22:59:04.214021 13066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9356702dbe4b21ba44b10a94f8dfb8. Latest consensus state: current_term: 1 leader_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } } }
I20260810 22:59:04.214248 13066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:04.214514 13076 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:59:04.217483 13076 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:59:04.222540 13076 catalog_manager.cc:1347] Generated new cluster ID: b2e33eda3bfb4fe283611b6b58d46ec0
I20260810 22:59:04.222604 13076 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:59:04.229846 13076 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:59:04.231060 13076 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:59:04.236626 13076 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Generated new TSK 0
I20260810 22:59:04.237352 13076 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:59:04.273072 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38533
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:04.382259 13089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:04.382498 13089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:04.382537 13089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:04.386276 13089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:04.386322 13089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:04.386376 13089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:59:04.391867 13089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:04.393158 13089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:04.394376 13089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:04.403767 13094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:04.403707 13097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:04.403883 13089 server_base.cc:1034] running on GCE node
W20260810 22:59:04.403803 13095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:04.404311 13089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:04.404906 13089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:04.406083 13089 hybrid_clock.cc:648] HybridClock initialized: now 1786402744406070 us; error 25 us; skew 500 ppm
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402744, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:04.408728 13089 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:59:04.409627 13089 webserver.cc:460] Webserver started at http://127.29.78.129:46199/ using document root <none> and password file <none>
I20260810 22:59:04.410190 13089 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:04.410249 13089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:04.410403 13089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:04.412256 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "9056f9a7aeaf4a08ae21373be93e85eb"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "15c7adefe4034bb3eeb229e07b78b79d"
server_key_iv: "0a4d9bfcd7f4c13553e4c7d14310e832"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.412776 13089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "9056f9a7aeaf4a08ae21373be93e85eb"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "15c7adefe4034bb3eeb229e07b78b79d"
server_key_iv: "0a4d9bfcd7f4c13553e4c7d14310e832"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.416605 13089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 22:59:04.418943 13106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.420269 13089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:04.420359 13089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "9056f9a7aeaf4a08ae21373be93e85eb"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "15c7adefe4034bb3eeb229e07b78b79d"
server_key_iv: "0a4d9bfcd7f4c13553e4c7d14310e832"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.420436 13089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:04.430425 13089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:04.433290 13089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:04.433429 13089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:04.434245 13089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:04.435216 13089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:04.435276 13089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.435322 13089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:04.435350 13089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.443111 13089 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:42985
I20260810 22:59:04.443188 13232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:42985 every 8 connection(s)
I20260810 22:59:04.443991 13089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:59:04.449615 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 13089
I20260810 22:59:04.449770 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:59:04.450079 30010 external_mini_cluster.cc:1428] Setting key 3fed87c5ce296199c49803ca51529db7
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402744, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:04.457486 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.445538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:46145 (local address 127.29.78.190:38533)
0810 22:59:04.445657 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.445660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.446446 (+ 786us) server_negotiation.cc:407] Connection header received
0810 22:59:04.447700 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.447703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.447765 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.447860 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:04.448987 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.449565 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.450930 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.451286 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.451390 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.454016 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:04.454021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:04.454026 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:04.454056 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:04.455394 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.455993 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.455995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.455996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.456062 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.456429 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.456432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.456433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.456609 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:04.456679 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.457289 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.457378 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 22:59:04.458755 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.445791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:46145)
0810 22:59:04.446299 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:04.446336 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:04.447487 (+ 1151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:04.448022 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:04.448028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:04.448387 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:04.448834 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.448848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.449685 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.449688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.450813 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.450824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.451456 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.451462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.451623 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.452221 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:04.452237 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:04.453856 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:04.455508 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:04.455516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:04.455531 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:04.455883 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:04.456145 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:04.456148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:04.456152 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:04.456330 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:04.456708 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:04.456716 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:04.457034 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 22:59:04.457248 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 22:59:04.457575 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":147,"threads_started":1}
I20260810 22:59:04.460036 13233 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38533
I20260810 22:59:04.460253 13233 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:59:04.460721 13233 heartbeater.cc:511] Master 127.29.78.190:38533 requested a full tablet report, sending...
I20260810 22:59:04.462191 13019 ts_manager.cc:194] Registered new tserver with Master: 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129:42985)
I20260810 22:59:04.463320 13019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:46145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:59:04.478207 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38533
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:04.586920 13238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:04.587191 13238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:04.587235 13238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:04.591150 13238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:04.591207 13238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:04.591264 13238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:59:04.596571 13238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:04.597587 13238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:04.598654 13238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:04.615154 13246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:04.615180 13243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:04.615438 13244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:04.616386 13238 server_base.cc:1034] running on GCE node
I20260810 22:59:04.616761 13238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:04.617373 13238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:04.618530 13238 hybrid_clock.cc:648] HybridClock initialized: now 1786402744618509 us; error 33 us; skew 500 ppm
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402744, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:04.621493 13238 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:59:04.622421 13238 webserver.cc:460] Webserver started at http://127.29.78.130:44835/ using document root <none> and password file <none>
I20260810 22:59:04.622927 13238 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:04.622990 13238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:04.623147 13238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:04.625100 13238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "b706fe118d1f45e799b92cfe47c1a268"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "0c132b4ccacaad88fb6b248e9d75d249"
server_key_iv: "047c15f6e4574f8cf7ecc9fcbd00df49"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.625605 13238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "b706fe118d1f45e799b92cfe47c1a268"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "0c132b4ccacaad88fb6b248e9d75d249"
server_key_iv: "047c15f6e4574f8cf7ecc9fcbd00df49"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.629230 13238 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:59:04.631642 13255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.632912 13238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:59:04.633049 13238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "b706fe118d1f45e799b92cfe47c1a268"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "0c132b4ccacaad88fb6b248e9d75d249"
server_key_iv: "047c15f6e4574f8cf7ecc9fcbd00df49"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.633154 13238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:04.647107 13238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:04.650401 13238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:04.650610 13238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:04.651146 13238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:04.651944 13238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:04.652019 13238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.652082 13238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:04.652127 13238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.660486 13238 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:46781
I20260810 22:59:04.660534 13382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:46781 every 8 connection(s)
I20260810 22:59:04.661278 13238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:59:04.664959 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 13238
I20260810 22:59:04.665104 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:59:04.665411 30010 external_mini_cluster.cc:1428] Setting key 26390166e0e087a2d1410ea4b75ff863
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402744, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:04.675005 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.662917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:59729 (local address 127.29.78.190:38533)
0810 22:59:04.663050 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.663054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.663850 (+ 796us) server_negotiation.cc:407] Connection header received
0810 22:59:04.665057 (+ 1207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.665061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.665119 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.665208 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:04.666581 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.667203 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.668351 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.668760 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.668872 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.671553 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:04.671557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:04.671559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:04.671596 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:04.673068 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.673536 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.673542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.673545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.673625 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.674007 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.674010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.674011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.674183 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:04.674282 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.674731 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.674848 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:59:04.675757 13385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.663298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:59729)
0810 22:59:04.663719 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:04.663756 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:04.664853 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:04.665376 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:04.665382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:04.665783 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:04.666420 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.666438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.667337 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.667340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.668209 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.668218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.668921 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.668924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.669066 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.669653 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:04.669668 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:04.671402 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:04.673186 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:04.673191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:04.673203 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:04.673440 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:04.673711 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:04.673715 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:04.673717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:04.673912 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:04.674280 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:04.674286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:04.674511 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 22:59:04.674675 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 22:59:04.674975 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":116,"threads_started":1}
I20260810 22:59:04.676779 13383 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38533
I20260810 22:59:04.677011 13383 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:04.677441 13383 heartbeater.cc:511] Master 127.29.78.190:38533 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260810 22:59:04.678555 13019 ts_manager.cc:194] Registered new tserver with Master: b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781)
I20260810 22:59:04.679132 13019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:59729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:59:04.692081 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38533
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:04.797294 13389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:04.797511 13389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:04.797549 13389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:04.801388 13389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:04.801433 13389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:04.801515 13389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:59:04.806017 13389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:04.806897 13389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:04.807816 13389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:04.817221 13398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:04.817221 13394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:04.817327 13389 server_base.cc:1034] running on GCE node
W20260810 22:59:04.817272 13395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:04.817935 13389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:04.818560 13389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:04.819717 13389 hybrid_clock.cc:648] HybridClock initialized: now 1786402744819690 us; error 39 us; skew 500 ppm
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402744, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:04.822413 13389 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:59:04.823340 13389 webserver.cc:460] Webserver started at http://127.29.78.131:33775/ using document root <none> and password file <none>
I20260810 22:59:04.823904 13389 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:04.823966 13389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:04.824160 13389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:04.826280 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "05a9c1232cf947c899b0bfe40397cc97"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cca2474ead123d3f2d1c910d5857fcba"
server_key_iv: "4705902cc21aefbcb9a68d14b2a5aaff"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.826745 13389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "05a9c1232cf947c899b0bfe40397cc97"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cca2474ead123d3f2d1c910d5857fcba"
server_key_iv: "4705902cc21aefbcb9a68d14b2a5aaff"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.830323 13389 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:59:04.832681 13405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.834040 13389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:04.834156 13389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "05a9c1232cf947c899b0bfe40397cc97"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cca2474ead123d3f2d1c910d5857fcba"
server_key_iv: "4705902cc21aefbcb9a68d14b2a5aaff"
server_key_version: "encryptionkey@0"
I20260810 22:59:04.834256 13389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:04.866391 13389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:04.869508 13389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:04.869745 13389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:04.870383 13389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:04.871266 13389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:04.871327 13389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.871381 13389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:04.871407 13389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:04.880843 13389 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:44995
I20260810 22:59:04.880887 13524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:44995 every 8 connection(s)
I20260810 22:59:04.881875 13389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:59:04.888172 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 13389
I20260810 22:59:04.888334 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:59:04.888618 30010 external_mini_cluster.cc:1428] Setting key e6886d64873817150736bb27727dd690
Aug 10 22:59:04 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402744, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:04.895562 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.883792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:49415 (local address 127.29.78.190:38533)
0810 22:59:04.883959 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.883963 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.884939 (+ 976us) server_negotiation.cc:407] Connection header received
0810 22:59:04.886029 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.886032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.886083 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.886159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:04.887325 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.887894 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.889299 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.889630 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.889737 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.892413 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:04.892416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:04.892418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:04.892442 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:04.893723 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.894262 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.894264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.894266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.894340 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.894634 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.894636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.894637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.894820 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:04.894893 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.895339 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.895435 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:59:04.896311 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.884163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:49415)
0810 22:59:04.884814 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:04.884852 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:04.885566 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:04.886318 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:04.886325 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:04.886673 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:04.887161 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.887174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.888035 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.888037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.889191 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.889201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.889773 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.889776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.890017 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.890654 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:04.890673 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:04.892242 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:04.893872 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:04.893879 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:04.893892 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:04.894175 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:04.894422 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:04.894423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:04.894425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:04.894550 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:04.894906 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:04.894910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:04.895134 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 22:59:04.895280 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 22:59:04.895509 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":128,"threads_started":1}
I20260810 22:59:04.897403 13525 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38533
I20260810 22:59:04.897629 13525 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:04.898141 13525 heartbeater.cc:511] Master 127.29.78.190:38533 requested a full tablet report, sending...
I20260810 22:59:04.899133 13019 ts_manager.cc:194] Registered new tserver with Master: 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
I20260810 22:59:04.899698 13019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:49415
I20260810 22:59:04.902508 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:59:04.911425 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.904394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.29.78.190:38533)
0810 22:59:04.904553 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.904557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.904694 (+ 137us) server_negotiation.cc:407] Connection header received
0810 22:59:04.904832 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.904835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.904874 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.904958 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:04.905669 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.906255 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.907070 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.907296 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.907403 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.908607 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:04.908610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:04.908613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:04.908637 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:04.909884 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.910279 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.910283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.910285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.910366 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:04.910620 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:04.910624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:04.910625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:04.910828 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:04.910961 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.911177 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.911283 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:59:04.914253 13019 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:59:04.917268 13019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:59:04.931730 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.925848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.29.78.129:42985)
0810 22:59:04.926178 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.926198 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.926811 (+ 613us) server_negotiation.cc:407] Connection header received
0810 22:59:04.926898 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.926903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.927115 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.927262 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:04.927842 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.929043 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.930466 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.930998 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.931119 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.931271 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.931445 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.931539 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":122,"threads_started":1}
I20260810 22:59:04.931720 13538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.925600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42985 (local address 127.0.0.1:57310)
0810 22:59:04.926483 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:04.926531 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:04.926672 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:04.927436 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:04.927442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:04.927474 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:04.927701 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.927709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.929200 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.929203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.930346 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.930359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.931129 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.931132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.931240 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.931285 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 22:59:04.931414 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 22:59:04.931539 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":60,"spinlock_wait_cycles":1280,"thread_start_us":63,"threads_started":1}
I20260810 22:59:04.932053 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.926426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57702 (local address 127.29.78.131:44995)
0810 22:59:04.926877 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.926888 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.926906 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:59:04.926962 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.926966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.927114 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.927262 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:04.927821 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.928708 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.930334 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.931134 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.931232 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.931359 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.931814 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.931905 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":159,"threads_started":1}
I20260810 22:59:04.932154 13536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.926004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:44995 (local address 127.0.0.1:57702)
0810 22:59:04.926621 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:04.926636 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:04.926724 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:04.927436 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:04.927442 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:04.927483 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:04.927697 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.927704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.928837 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.928843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.930196 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.930206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.931420 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.931424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.931521 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.931550 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:59:04.931661 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 22:59:04.931731 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":35,"spinlock_wait_cycles":3456,"thread_start_us":93,"threads_started":1}
I20260810 22:59:04.932500 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.926311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46781 (local address 127.0.0.1:43262)
0810 22:59:04.927603 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:04.927616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:04.927709 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:04.928152 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:04.928155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:04.928174 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:04.928246 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.928251 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.929638 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.929641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.930768 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:04.930779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.932047 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.932053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:04.932166 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.932193 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:59:04.932268 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 22:59:04.932333 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260810 22:59:04.932979 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:04.926481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.29.78.130:46781)
0810 22:59:04.927069 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:04.927080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:04.927715 (+ 635us) server_negotiation.cc:407] Connection header received
0810 22:59:04.927803 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:04.927807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:04.927990 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:04.928154 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:04.928379 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.929403 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.930917 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:04.931745 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:04.931845 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:04.931968 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:04.932713 (+ 745us) server_negotiation.cc:299] Negotiation successful
0810 22:59:04.932817 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":109,"threads_started":1}
I20260810 22:59:04.935184 13310 tablet_service.cc:1467] Processing CreateTablet for tablet ae6aa76e442c4e23926890332e25db08 (DEFAULT_TABLE table=test-table [id=c486b7b18d0e4bc082792716cd9ae279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:04.935196 13457 tablet_service.cc:1467] Processing CreateTablet for tablet ae6aa76e442c4e23926890332e25db08 (DEFAULT_TABLE table=test-table [id=c486b7b18d0e4bc082792716cd9ae279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:04.935196 13156 tablet_service.cc:1467] Processing CreateTablet for tablet ae6aa76e442c4e23926890332e25db08 (DEFAULT_TABLE table=test-table [id=c486b7b18d0e4bc082792716cd9ae279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:04.936265 13310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6aa76e442c4e23926890332e25db08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:04.936275 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6aa76e442c4e23926890332e25db08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:04.936283 13457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6aa76e442c4e23926890332e25db08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:04.942865 13542 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Bootstrap starting.
I20260810 22:59:04.944331 13545 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap starting.
I20260810 22:59:04.945536 13542 tablet_bootstrap.cc:654] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:04.946174 13545 tablet_bootstrap.cc:654] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:04.946163 13543 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Bootstrap starting.
I20260810 22:59:04.946401 13542 log.cc:824] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:04.947400 13545 log.cc:824] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:04.948553 13543 tablet_bootstrap.cc:654] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:04.948642 13542 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: No bootstrap required, opened a new log
I20260810 22:59:04.948824 13542 ts_tablet_manager.cc:1397] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 22:59:04.949334 13543 log.cc:824] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:04.949447 13545 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: No bootstrap required, opened a new log
I20260810 22:59:04.949627 13545 ts_tablet_manager.cc:1397] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:59:04.951324 13543 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: No bootstrap required, opened a new log
I20260810 22:59:04.951511 13543 ts_tablet_manager.cc:1397] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:59:04.952078 13542 raft_consensus.cc:348] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:04.952312 13542 raft_consensus.cc:374] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:04.952364 13542 raft_consensus.cc:729] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706fe118d1f45e799b92cfe47c1a268, State: Initialized, Role: FOLLOWER
I20260810 22:59:04.952804 13542 consensus_queue.cc:260] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:04.953418 13383 heartbeater.cc:503] Master 127.29.78.190:38533 was elected leader, sending a full tablet report...
I20260810 22:59:04.953382 13545 raft_consensus.cc:348] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:04.953581 13545 raft_consensus.cc:374] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:04.953629 13545 raft_consensus.cc:729] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9056f9a7aeaf4a08ae21373be93e85eb, State: Initialized, Role: FOLLOWER
I20260810 22:59:04.953732 13542 ts_tablet_manager.cc:1428] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260810 22:59:04.954026 13545 consensus_queue.cc:260] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:04.954941 13233 heartbeater.cc:503] Master 127.29.78.190:38533 was elected leader, sending a full tablet report...
I20260810 22:59:04.954938 13545 ts_tablet_manager.cc:1428] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260810 22:59:04.955096 13543 raft_consensus.cc:348] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:04.955307 13543 raft_consensus.cc:374] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:04.955344 13543 raft_consensus.cc:729] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a9c1232cf947c899b0bfe40397cc97, State: Initialized, Role: FOLLOWER
I20260810 22:59:04.955628 13543 consensus_queue.cc:260] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:04.956373 13543 ts_tablet_manager.cc:1428] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:59:04.956691 13525 heartbeater.cc:503] Master 127.29.78.190:38533 was elected leader, sending a full tablet report...
W20260810 22:59:05.132882 13526 tablet.cc:2369] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:05.157523 13551 raft_consensus.cc:482] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:05.157902 13551 raft_consensus.cc:504] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:05.159044 13551 leader_election.cc:288] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129:42985), b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781)
W20260810 22:59:05.162369 13384 tablet.cc:2369] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:05.162638 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.159284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42985 (local address 127.29.78.131:41911)
0810 22:59:05.159432 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:05.159449 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:05.159575 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:05.159913 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:05.159916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:05.159936 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:05.160044 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.160049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.160757 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.160761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.161473 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.161481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.162188 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.162191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.162313 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.162337 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 22:59:05.162421 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 22:59:05.162507 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 22:59:05.162693 13537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.159439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:41911 (local address 127.29.78.129:42985)
0810 22:59:05.159610 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:05.159615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:05.159650 (+ 35us) server_negotiation.cc:407] Connection header received
0810 22:59:05.159711 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:05.159715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:05.159776 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:05.159855 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:05.160157 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.160647 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.161594 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.162050 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.162172 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.162221 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:05.162451 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 22:59:05.162524 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 22:59:05.163236 13552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.159614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46781 (local address 127.29.78.131:49669)
0810 22:59:05.159916 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:05.159930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:05.160064 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:05.160338 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:05.160341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:05.160354 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:05.160474 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.160479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.161129 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.161132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.161874 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.161883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.162839 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.162844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.162949 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.162975 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 22:59:05.163055 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 22:59:05.163116 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":117,"threads_started":1}
I20260810 22:59:05.163256 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.159785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:49669 (local address 127.29.78.130:46781)
0810 22:59:05.159928 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:05.159933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:05.160025 (+ 92us) server_negotiation.cc:407] Connection header received
0810 22:59:05.160175 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:05.160177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:05.160233 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:05.160328 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:05.160573 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.161023 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.162028 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.162562 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.162696 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.162754 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:05.163070 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 22:59:05.163151 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 22:59:05.163252 13178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6aa76e442c4e23926890332e25db08" candidate_uuid: "05a9c1232cf947c899b0bfe40397cc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" is_pre_election: true
I20260810 22:59:05.163573 13178 raft_consensus.cc:2454] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a9c1232cf947c899b0bfe40397cc97 in term 0.
I20260810 22:59:05.163957 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6aa76e442c4e23926890332e25db08" candidate_uuid: "05a9c1232cf947c899b0bfe40397cc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706fe118d1f45e799b92cfe47c1a268" is_pre_election: true
I20260810 22:59:05.164062 13407 leader_election.cc:302] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a9c1232cf947c899b0bfe40397cc97, 9056f9a7aeaf4a08ae21373be93e85eb; no voters:
I20260810 22:59:05.164197 13331 raft_consensus.cc:2454] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05a9c1232cf947c899b0bfe40397cc97 in term 0.
I20260810 22:59:05.164367 13551 raft_consensus.cc:2781] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:59:05.164463 13551 raft_consensus.cc:482] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:59:05.164548 13551 raft_consensus.cc:3037] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:05.165542 13551 raft_consensus.cc:504] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:05.165879 13551 leader_election.cc:288] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [CANDIDATE]: Term 1 election: Requested vote from peers 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129:42985), b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781)
I20260810 22:59:05.166215 13178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6aa76e442c4e23926890332e25db08" candidate_uuid: "05a9c1232cf947c899b0bfe40397cc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9056f9a7aeaf4a08ae21373be93e85eb"
I20260810 22:59:05.166311 13331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6aa76e442c4e23926890332e25db08" candidate_uuid: "05a9c1232cf947c899b0bfe40397cc97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706fe118d1f45e799b92cfe47c1a268"
I20260810 22:59:05.166329 13178 raft_consensus.cc:3037] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:05.166429 13331 raft_consensus.cc:3037] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:05.167356 13178 raft_consensus.cc:2454] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a9c1232cf947c899b0bfe40397cc97 in term 1.
I20260810 22:59:05.167447 13331 raft_consensus.cc:2454] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05a9c1232cf947c899b0bfe40397cc97 in term 1.
I20260810 22:59:05.167714 13407 leader_election.cc:302] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a9c1232cf947c899b0bfe40397cc97, b706fe118d1f45e799b92cfe47c1a268; no voters:
I20260810 22:59:05.167934 13551 raft_consensus.cc:2781] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:05.168133 13551 raft_consensus.cc:686] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 LEADER]: Becoming Leader. State: Replica: 05a9c1232cf947c899b0bfe40397cc97, State: Running, Role: LEADER
I20260810 22:59:05.168470 13551 consensus_queue.cc:237] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:05.171612 13018 catalog_manager.cc:5495] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131). New cstate: current_term: 1 leader_uuid: "05a9c1232cf947c899b0bfe40397cc97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } health_report { overall_health: HEALTHY } } }
W20260810 22:59:05.195245 13234 tablet.cc:2369] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:05.222551 13540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.219076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.29.78.131:44995)
0810 22:59:05.219224 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:05.219229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:05.219245 (+ 16us) server_negotiation.cc:407] Connection header received
0810 22:59:05.219391 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:05.219393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:05.219448 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:05.219573 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:59:05.219818 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.220287 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.221094 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.221304 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.221409 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.221732 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:59:05.222178 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:59:05.222305 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:05.222355 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 22:59:05.222412 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 22:59:05.228430 13178 raft_consensus.cc:1261] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Refusing update from remote peer 05a9c1232cf947c899b0bfe40397cc97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:05.228428 13331 raft_consensus.cc:1261] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Refusing update from remote peer 05a9c1232cf947c899b0bfe40397cc97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:05.229189 13553 consensus_queue.cc:1035] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:05.229373 13551 consensus_queue.cc:1035] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:05.236796 13556 mvcc.cc:205] Tried to move back new op lower bound from 7317105644447911936 to 7317105644212641792. Current Snapshot: MvccSnapshot[applied={T|T < 7317105644447911936}]
I20260810 22:59:05.237776 13557 mvcc.cc:205] Tried to move back new op lower bound from 7317105644447911936 to 7317105644212641792. Current Snapshot: MvccSnapshot[applied={T|T < 7317105644447911936}]
I20260810 22:59:05.244661 13017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:05.244838 13017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:05.247231 13017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Sending DeleteTablet for 3 replicas of tablet ae6aa76e442c4e23926890332e25db08
I20260810 22:59:05.248049 13457 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6aa76e442c4e23926890332e25db08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:05 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57702
I20260810 22:59:05.248142 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 13089
I20260810 22:59:05.248109 13310 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6aa76e442c4e23926890332e25db08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:05 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:43262
I20260810 22:59:05.248096 13156 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6aa76e442c4e23926890332e25db08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:05 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57310
I20260810 22:59:05.250841 13565 tablet_replica.cc:330] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: stopping tablet replica
I20260810 22:59:05.251173 13565 raft_consensus.cc:2229] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:59:05.251583 13565 raft_consensus.cc:2258] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:05.251971 13566 tablet_replica.cc:330] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: stopping tablet replica
I20260810 22:59:05.252243 13566 raft_consensus.cc:2229] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:05.252549 13566 raft_consensus.cc:2258] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:05.253958 13565 ts_tablet_manager.cc:1905] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:05.254199 13566 ts_tablet_manager.cc:1905] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:05.256784 13565 ts_tablet_manager.cc:1918] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:05.256883 13565 log.cc:1196] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/ae6aa76e442c4e23926890332e25db08
I20260810 22:59:05.257197 13565 ts_tablet_manager.cc:1939] T ae6aa76e442c4e23926890332e25db08 P 05a9c1232cf947c899b0bfe40397cc97: Deleting consensus metadata
I20260810 22:59:05.257597 13566 ts_tablet_manager.cc:1918] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:05.257730 13566 log.cc:1196] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/ae6aa76e442c4e23926890332e25db08
I20260810 22:59:05.258076 13566 ts_tablet_manager.cc:1939] T ae6aa76e442c4e23926890332e25db08 P b706fe118d1f45e799b92cfe47c1a268: Deleting consensus metadata
I20260810 22:59:05.258298 13006 catalog_manager.cc:4841] TS 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995): tablet ae6aa76e442c4e23926890332e25db08 (table test-table [id=c486b7b18d0e4bc082792716cd9ae279]) successfully deleted
I20260810 22:59:05.259310 13004 catalog_manager.cc:4841] TS b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781): tablet ae6aa76e442c4e23926890332e25db08 (table test-table [id=c486b7b18d0e4bc082792716cd9ae279]) successfully deleted
W20260810 22:59:05.259635 13004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:42985 (error 108)
I20260810 22:59:05.259842 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 13238
I20260810 22:59:05.267582 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.267311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42985 (local address 127.0.0.1:57316)
0810 22:59:05.267438 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:05.267510 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 22:59:05.267817 13004 catalog_manager.cc:4584] TS 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129:42985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae6aa76e442c4e23926890332e25db08: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42985: connect: Connection refused (error 111)
I20260810 22:59:05.273545 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 13389
I20260810 22:59:05.281728 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 12984
I20260810 22:59:05.289402 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38533
--webserver_interface=127.29.78.190
--webserver_port=45669
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:05.395792 13568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:05.396117 13568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:05.396214 13568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:05.400555 13568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:59:05.400609 13568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:05.400628 13568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:05.400643 13568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:59:05.400699 13568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:59:05.405517 13568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:05.406471 13568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:05.407382 13568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:05.421195 13577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:05.421205 13576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:05.421299 13568 server_base.cc:1034] running on GCE node
W20260810 22:59:05.421195 13579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:05.422122 13568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:05.423465 13568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:05.424659 13568 hybrid_clock.cc:648] HybridClock initialized: now 1786402745424628 us; error 53 us; skew 500 ppm
Aug 10 22:59:05 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:05.427452 13568 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:59:05.428378 13568 webserver.cc:460] Webserver started at http://127.29.78.190:45669/ using document root <none> and password file <none>
I20260810 22:59:05.428969 13568 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:05.429077 13568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:05.432900 13568 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260810 22:59:05.435274 13586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:05.436437 13568 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 22:59:05.436574 13568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "ea9356702dbe4b21ba44b10a94f8dfb8"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cc5ab17e7955180780ae4100713d4db9"
server_key_iv: "8a544753c146507babdf8413611c1266"
server_key_version: "encryptionkey@0"
I20260810 22:59:05.437186 13568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:05.456072 13568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:05.459285 13568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:05.459494 13568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:05.466261 13568 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:38533
I20260810 22:59:05.466544 13641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:38533 every 8 connection(s)
I20260810 22:59:05.467361 13568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:59:05.471848 13642 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:59:05.473403 13642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Bootstrap starting.
I20260810 22:59:05.476159 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 13568
I20260810 22:59:05.476575 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:42985
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:38533
--webserver_port=46199
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:59:05.490583 13642 log.cc:824] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:05.499332 13642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:59:05.499750 13642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Bootstrap complete.
I20260810 22:59:05.503871 13642 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:05.504281 13642 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9356702dbe4b21ba44b10a94f8dfb8, State: Initialized, Role: FOLLOWER
I20260810 22:59:05.504720 13642 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:05.504856 13642 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:59:05.504949 13642 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:59:05.505057 13642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:59:05.506829 13642 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:05.507138 13642 leader_election.cc:302] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9356702dbe4b21ba44b10a94f8dfb8; no voters:
I20260810 22:59:05.507455 13642 leader_election.cc:288] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:59:05.507531 13647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:59:05.507696 13647 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 LEADER]: Becoming Leader. State: Replica: ea9356702dbe4b21ba44b10a94f8dfb8, State: Running, Role: LEADER
I20260810 22:59:05.507967 13647 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:05.508584 13642 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:05.508812 13650 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9356702dbe4b21ba44b10a94f8dfb8. Latest consensus state: current_term: 2 leader_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } } }
I20260810 22:59:05.508939 13650 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:05.509352 13652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:59:05.509613 13648 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } } }
I20260810 22:59:05.509704 13648 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:05.513969 13652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:59:05.514537 13652 catalog_manager.cc:1259] Loaded cluster ID: b2e33eda3bfb4fe283611b6b58d46ec0
I20260810 22:59:05.514595 13652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:59:05.515898 13652 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 22:59:05.520536 13652 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 22:59:05.525447 13652 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:59:05.526110 13652 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Loaded TSK: 0
I20260810 22:59:05.527172 13652 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:59:05.588485 13645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:05.588759 13645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:05.588827 13645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:05.592986 13645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:05.593046 13645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:05.593101 13645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:59:05.598428 13645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:42985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=46199
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:05.599455 13645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:05.600625 13645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:05.611303 13673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:05.611373 13674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:05.611393 13676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:05.611785 13645 server_base.cc:1034] running on GCE node
I20260810 22:59:05.612223 13645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:05.612891 13645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:05.614084 13645 hybrid_clock.cc:648] HybridClock initialized: now 1786402745614064 us; error 35 us; skew 500 ppm
Aug 10 22:59:05 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:05.616735 13645 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:59:05.617708 13645 webserver.cc:460] Webserver started at http://127.29.78.129:46199/ using document root <none> and password file <none>
I20260810 22:59:05.618284 13645 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:05.618393 13645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:05.622124 13645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 22:59:05.624325 13684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:05.625543 13645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:05.625667 13645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "9056f9a7aeaf4a08ae21373be93e85eb"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "15c7adefe4034bb3eeb229e07b78b79d"
server_key_iv: "0a4d9bfcd7f4c13553e4c7d14310e832"
server_key_version: "encryptionkey@0"
I20260810 22:59:05.626237 13645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:05.638306 13645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:05.641390 13645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:05.641572 13645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:05.642158 13645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:05.643303 13691 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:59:05.647187 13645 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:59:05.647248 13645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260810 22:59:05.647312 13645 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:59:05.648939 13645 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:59:05.649006 13645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:59:05.649250 13691 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap starting.
I20260810 22:59:05.659704 13645 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:42985
I20260810 22:59:05.659781 13805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:42985 every 8 connection(s)
I20260810 22:59:05.660815 13645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:59:05.663548 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 13645
I20260810 22:59:05.664021 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:46781
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:38533
--webserver_port=44835
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:59:05.669685 13691 log.cc:824] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:59:05 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:05.677663 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.662993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:34025 (local address 127.29.78.190:38533)
0810 22:59:05.665147 (+ 2154us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:05.665168 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:05.665211 (+ 43us) server_negotiation.cc:407] Connection header received
0810 22:59:05.665458 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:05.665472 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:05.665861 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:05.666192 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:05.667268 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.668196 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.669345 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.670173 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.670338 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.672985 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:05.672991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:05.673008 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:05.673052 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:05.674732 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:05.675242 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:05.675247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:05.675251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:05.675364 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:05.675705 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:05.675709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:05.675711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:05.676026 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:05.676153 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:05.676657 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 22:59:05.676895 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260810 22:59:05.677675 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.663022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:34025)
0810 22:59:05.663556 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:05.663612 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:05.664648 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:05.666236 (+ 1588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:05.666244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:05.666643 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:05.667091 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.667103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.668362 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.668364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.669213 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.669222 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.670341 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.670345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.670534 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.671181 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:05.671196 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:05.672781 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:05.674852 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:05.674857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:05.674868 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:05.675125 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:05.675459 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:05.675461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:05.675463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:05.675613 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:05.676143 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:05.676148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:05.676370 (+ 222us) client_negotiation.cc:769] Sending connection context
0810 22:59:05.676539 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 22:59:05.676844 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":149,"threads_started":1}
I20260810 22:59:05.679052 13691 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:59:05.679481 13691 tablet_bootstrap.cc:492] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap complete.
I20260810 22:59:05.680119 13806 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38533
I20260810 22:59:05.680342 13806 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:05.680809 13806 heartbeater.cc:511] Master 127.29.78.190:38533 requested a full tablet report, sending...
I20260810 22:59:05.680996 13691 ts_tablet_manager.cc:1397] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.004s
I20260810 22:59:05.683331 13605 ts_manager.cc:194] Registered new tserver with Master: 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129:42985)
I20260810 22:59:05.683564 13691 raft_consensus.cc:348] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:05.684006 13691 raft_consensus.cc:729] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9056f9a7aeaf4a08ae21373be93e85eb, State: Initialized, Role: FOLLOWER
I20260810 22:59:05.684463 13691 consensus_queue.cc:260] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } } peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } }
I20260810 22:59:05.685391 13691 ts_tablet_manager.cc:1428] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20260810 22:59:05.686710 13605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:34025
I20260810 22:59:05.689069 13806 heartbeater.cc:503] Master 127.29.78.190:38533 was elected leader, sending a full tablet report...
Aug 10 22:59:05 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:59:05.697575 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.686302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42985 (local address 127.0.0.1:57318)
0810 22:59:05.687237 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:05.687278 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:05.687393 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:05.687991 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:05.687997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:05.688401 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:05.688562 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.688570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.690003 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.690008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.690984 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.690993 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.691778 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.691780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.691854 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.692374 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:05.692386 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:05.693880 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:05.696025 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:05.696033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:05.696036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:05.696252 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:05.696555 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:05.696557 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:05.696558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:05.696631 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:05.697115 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:05.697119 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:05.697208 (+ 89us) client_negotiation.cc:769] Sending connection context
0810 22:59:05.697310 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 22:59:05.697444 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":104,"threads_started":1}
I20260810 22:59:05.697731 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.686338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.29.78.129:42985)
0810 22:59:05.687007 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:05.687022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:05.687530 (+ 508us) server_negotiation.cc:407] Connection header received
0810 22:59:05.687601 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:05.687606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:05.687834 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:05.687963 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:05.688692 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.689886 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.691098 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.691662 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.691778 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.694020 (+ 2242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:05.694030 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:05.694040 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:05.694065 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:05.695917 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:05.696358 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:05.696364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:05.696366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:05.696463 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:05.696714 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:05.696717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:05.696719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:05.697006 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:05.697125 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:05.697413 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 22:59:05.697555 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"spinlock_wait_cycles":2816,"thread_start_us":90,"threads_started":1}
I20260810 22:59:05.698594 13734 tablet_service.cc:1514] Processing DeleteTablet for tablet ae6aa76e442c4e23926890332e25db08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:05 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57318
I20260810 22:59:05.699572 13815 tablet_replica.cc:330] stopping tablet replica
I20260810 22:59:05.699858 13815 raft_consensus.cc:2229] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:05.700098 13815 raft_consensus.cc:2258] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:05.700987 13815 ts_tablet_manager.cc:1905] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:05.704553 13815 ts_tablet_manager.cc:1918] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:05.704632 13815 log.cc:1196] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/ae6aa76e442c4e23926890332e25db08
I20260810 22:59:05.704909 13815 ts_tablet_manager.cc:1939] T ae6aa76e442c4e23926890332e25db08 P 9056f9a7aeaf4a08ae21373be93e85eb: Deleting consensus metadata
I20260810 22:59:05.706658 13589 catalog_manager.cc:4841] TS 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129:42985): tablet ae6aa76e442c4e23926890332e25db08 (table test-table [id=c486b7b18d0e4bc082792716cd9ae279]) successfully deleted
W20260810 22:59:05.788770 13810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:05.789067 13810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:05.789141 13810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:05.793030 13810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:05.793112 13810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:05.793200 13810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:59:05.798219 13810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:46781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=44835
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:05.799237 13810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:05.800220 13810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:05.810107 13823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:05.810108 13822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:05.810211 13810 server_base.cc:1034] running on GCE node
W20260810 22:59:05.810371 13825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:05.810709 13810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:05.811247 13810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:05.812415 13810 hybrid_clock.cc:648] HybridClock initialized: now 1786402745812400 us; error 26 us; skew 500 ppm
Aug 10 22:59:05 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:05.815048 13810 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:59:05.815882 13810 webserver.cc:460] Webserver started at http://127.29.78.130:44835/ using document root <none> and password file <none>
I20260810 22:59:05.816406 13810 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:05.816496 13810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:05.820158 13810 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:05.822067 13832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:05.823225 13810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:05.823354 13810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "b706fe118d1f45e799b92cfe47c1a268"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "0c132b4ccacaad88fb6b248e9d75d249"
server_key_iv: "047c15f6e4574f8cf7ecc9fcbd00df49"
server_key_version: "encryptionkey@0"
I20260810 22:59:05.823887 13810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:05.846558 13810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:05.849184 13810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:05.849380 13810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:05.849977 13810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:05.850750 13810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:05.850818 13810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:05.850883 13810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:05.850921 13810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:05.859189 13810 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:46781
I20260810 22:59:05.859217 13954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:46781 every 8 connection(s)
I20260810 22:59:05.860147 13810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:59:05.864159 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 13810
I20260810 22:59:05.864548 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:44995
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:38533
--webserver_port=33775
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:59:05 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:05.873643 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.861800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:53591 (local address 127.29.78.190:38533)
0810 22:59:05.861973 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:05.861977 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:05.862681 (+ 704us) server_negotiation.cc:407] Connection header received
0810 22:59:05.863954 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:05.863957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:05.864029 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:05.864137 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:05.865298 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.865794 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.866944 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.867177 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.867288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.870072 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:05.870076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:05.870078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:05.870103 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:05.871577 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:05.872127 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:05.872131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:05.872133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:05.872217 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:05.872566 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:05.872571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:05.872573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:05.872755 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:05.872866 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:05.873333 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 22:59:05.873473 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 22:59:05.874459 13959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:05.862105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:53591)
0810 22:59:05.862541 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:05.862580 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:05.863740 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:05.864333 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:05.864339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:05.864704 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:05.865161 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.865173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.865976 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.865980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.866817 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:05.866826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:05.867331 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:05.867336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:05.867501 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:05.868104 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:05.868119 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:05.869891 (+ 1772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:05.871741 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:05.871746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:05.871759 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:05.872018 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:05.872313 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:05.872317 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:05.872319 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:05.872467 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:05.872865 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:05.872871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:05.873110 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 22:59:05.873292 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 22:59:05.873599 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":102,"threads_started":1}
I20260810 22:59:05.875627 13956 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38533
I20260810 22:59:05.875860 13956 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:05.876317 13956 heartbeater.cc:511] Master 127.29.78.190:38533 requested a full tablet report, sending...
I20260810 22:59:05.877321 13605 ts_manager.cc:194] Registered new tserver with Master: b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781)
I20260810 22:59:05.878100 13605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:53591
W20260810 22:59:05.971776 13960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:05.972002 13960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:05.972040 13960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:05.976193 13960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:05.976243 13960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:05.976292 13960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:59:05.981060 13960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:44995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=33775
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:05.981940 13960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:05.982873 13960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:05.992094 13971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:05.992147 13960 server_base.cc:1034] running on GCE node
W20260810 22:59:05.992097 13968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:05.992156 13966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:05.992741 13960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:05.993283 13960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:05.994433 13960 hybrid_clock.cc:648] HybridClock initialized: now 1786402745994418 us; error 32 us; skew 500 ppm
Aug 10 22:59:05 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:05.997056 13960 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:59:05.997799 13960 webserver.cc:460] Webserver started at http://127.29.78.131:33775/ using document root <none> and password file <none>
I20260810 22:59:05.998364 13960 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:05.998508 13960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:06.002067 13960 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 22:59:06.003976 13978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:06.005122 13960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:06.005272 13960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "05a9c1232cf947c899b0bfe40397cc97"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cca2474ead123d3f2d1c910d5857fcba"
server_key_iv: "4705902cc21aefbcb9a68d14b2a5aaff"
server_key_version: "encryptionkey@0"
I20260810 22:59:06.005770 13960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:06.032083 13960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:06.035386 13960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:06.035597 13960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:06.036199 13960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:06.036995 13960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:06.037081 13960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:06.037139 13960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:06.037184 13960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:06.046435 13960 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:44995
I20260810 22:59:06.046464 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:44995 every 8 connection(s)
I20260810 22:59:06.047384 13960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:59:06.051422 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 13960
Aug 10 22:59:06 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402745, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:06.061100 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.052098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.29.78.190:38533)
0810 22:59:06.052347 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.052353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.052484 (+ 131us) server_negotiation.cc:407] Connection header received
0810 22:59:06.052595 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.052598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.052641 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.052727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:06.053414 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.054186 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.055332 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.055559 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.055681 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.057341 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:06.057344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:06.057346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:06.057369 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:06.059171 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:06.059700 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:06.059704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:06.059708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:06.059833 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:06.060149 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:06.060152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:06.060153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:06.060414 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:06.060619 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.060806 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.060930 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260810 22:59:06.063313 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.049157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:59523 (local address 127.29.78.190:38533)
0810 22:59:06.049295 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.049300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.050199 (+ 899us) server_negotiation.cc:407] Connection header received
0810 22:59:06.051625 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.051629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.051689 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.051797 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:06.053234 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.054115 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.055618 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.055882 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.055952 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.059347 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:06.059354 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:06.059356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:06.059385 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:06.061180 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:06.061780 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:06.061783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:06.061784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:06.061916 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:06.062344 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:06.062347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:06.062348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:06.062542 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:06.062636 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.063136 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.063218 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":55}
I20260810 22:59:06.064128 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.049537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:59523)
0810 22:59:06.050050 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.050095 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:06.051355 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:06.052005 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:06.052015 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:06.052524 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:06.053087 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.053101 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.054250 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.054254 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.055510 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.055520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.056373 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.056377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.056612 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.057169 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:06.057189 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:06.059178 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:06.061302 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:06.061307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:06.061319 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:06.061688 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:06.062030 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:06.062033 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:06.062034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:06.062239 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:06.062639 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:06.062643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:06.062879 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 22:59:06.063095 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 22:59:06.063336 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":122,"threads_started":1}
I20260810 22:59:06.065070 14107 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38533
I20260810 22:59:06.065286 14107 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:06.065724 14107 heartbeater.cc:511] Master 127.29.78.190:38533 requested a full tablet report, sending...
I20260810 22:59:06.066684 13605 ts_manager.cc:194] Registered new tserver with Master: 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
I20260810 22:59:06.067198 13605 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:59523
I20260810 22:59:06.074477 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:59:06.078327 13808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.075079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51726 (local address 127.29.78.190:38533)
0810 22:59:06.075210 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.075214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.075351 (+ 137us) server_negotiation.cc:407] Connection header received
0810 22:59:06.075490 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.075492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.075580 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.075663 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:59:06.075824 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.076275 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.077064 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.077256 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.077327 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.077578 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:59:06.077992 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:59:06.078078 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.078136 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.078175 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:59:06.078922 13605 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:59:06.082201 13605 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:59:06.089962 13734 tablet_service.cc:1467] Processing CreateTablet for tablet dc55b83098954754a4b47eac960517a5 (DEFAULT_TABLE table=test-table [id=27c7e491dd334df6a35bd5ba90caf723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:06.090691 13734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc55b83098954754a4b47eac960517a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:06.093447 13691 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap starting.
I20260810 22:59:06.094367 13691 tablet_bootstrap.cc:654] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:06.094485 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.088827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46781 (local address 127.0.0.1:43268)
0810 22:59:06.088945 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.088963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:06.089043 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:06.089933 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:06.089937 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:06.089956 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:06.090065 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.090073 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.091614 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.091617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.092798 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.092807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.094068 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.094072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.094163 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.094203 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 22:59:06.094277 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 22:59:06.094354 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":22}
I20260810 22:59:06.094539 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.089051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.29.78.130:46781)
0810 22:59:06.089430 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.089442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.089466 (+ 24us) server_negotiation.cc:407] Connection header received
0810 22:59:06.089544 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.089549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.089759 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.089947 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:06.090298 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.091457 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.092916 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.093944 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.094068 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.094210 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.094316 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.094406 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":146,"threads_started":1}
I20260810 22:59:06.095813 13691 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: No bootstrap required, opened a new log
I20260810 22:59:06.095901 13691 ts_tablet_manager.cc:1397] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:06.095940 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.089712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57720 (local address 127.29.78.131:44995)
0810 22:59:06.090222 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.090233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.090279 (+ 46us) server_negotiation.cc:407] Connection header received
0810 22:59:06.090348 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.090371 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.090747 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.090899 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:06.091291 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.092363 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.093874 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.095155 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.095269 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.095400 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.095719 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.095814 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":199,"threads_started":1}
I20260810 22:59:06.095973 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.089293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:44995 (local address 127.0.0.1:57720)
0810 22:59:06.089520 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.089535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:06.089626 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:06.091063 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:06.091066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:06.091085 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:06.091169 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.091174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.092519 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.092522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.093698 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.093710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.095457 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.095460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.095543 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.095566 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:59:06.095688 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 22:59:06.095756 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":20,"thread_start_us":93,"threads_started":1}
I20260810 22:59:06.096374 13691 raft_consensus.cc:348] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.096486 13691 raft_consensus.cc:374] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:06.096524 13691 raft_consensus.cc:729] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9056f9a7aeaf4a08ae21373be93e85eb, State: Initialized, Role: FOLLOWER
I20260810 22:59:06.096657 13691 consensus_queue.cc:260] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.096916 13691 ts_tablet_manager.cc:1428] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:59:06.097602 13885 tablet_service.cc:1467] Processing CreateTablet for tablet dc55b83098954754a4b47eac960517a5 (DEFAULT_TABLE table=test-table [id=27c7e491dd334df6a35bd5ba90caf723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:06.098683 13885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc55b83098954754a4b47eac960517a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:06.098762 14034 tablet_service.cc:1467] Processing CreateTablet for tablet dc55b83098954754a4b47eac960517a5 (DEFAULT_TABLE table=test-table [id=27c7e491dd334df6a35bd5ba90caf723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:06.099668 14034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc55b83098954754a4b47eac960517a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:06.105383 14120 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Bootstrap starting.
I20260810 22:59:06.105695 14121 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Bootstrap starting.
I20260810 22:59:06.107694 14120 tablet_bootstrap.cc:654] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:06.108183 14121 tablet_bootstrap.cc:654] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:06.108474 14120 log.cc:824] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:06.108927 14121 log.cc:824] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:06.110371 14120 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: No bootstrap required, opened a new log
I20260810 22:59:06.110520 14120 ts_tablet_manager.cc:1397] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:59:06.110949 14121 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: No bootstrap required, opened a new log
I20260810 22:59:06.111130 14121 ts_tablet_manager.cc:1397] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:59:06.113561 14120 raft_consensus.cc:348] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.113749 14120 raft_consensus.cc:374] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:06.113811 14120 raft_consensus.cc:729] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b706fe118d1f45e799b92cfe47c1a268, State: Initialized, Role: FOLLOWER
I20260810 22:59:06.114107 14121 raft_consensus.cc:348] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.114230 14120 consensus_queue.cc:260] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.114301 14121 raft_consensus.cc:374] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:06.114360 14121 raft_consensus.cc:729] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05a9c1232cf947c899b0bfe40397cc97, State: Initialized, Role: FOLLOWER
I20260810 22:59:06.114712 14121 consensus_queue.cc:260] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.114804 13956 heartbeater.cc:503] Master 127.29.78.190:38533 was elected leader, sending a full tablet report...
I20260810 22:59:06.115339 14107 heartbeater.cc:503] Master 127.29.78.190:38533 was elected leader, sending a full tablet report...
I20260810 22:59:06.115583 14121 ts_tablet_manager.cc:1428] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 22:59:06.116102 14120 ts_tablet_manager.cc:1428] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 22:59:06.162235 13807 tablet.cc:2369] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 22:59:06.298710 14108 tablet.cc:2369] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:06.330883 13812 raft_consensus.cc:482] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:06.331157 13812 raft_consensus.cc:504] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.332413 13812 leader_election.cc:288] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781), 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
I20260810 22:59:06.336256 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.332886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:59661 (local address 127.29.78.131:44995)
0810 22:59:06.333043 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.333047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.333143 (+ 96us) server_negotiation.cc:407] Connection header received
0810 22:59:06.333303 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.333305 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.333344 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.333418 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:06.333734 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.334250 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.335332 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.335703 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.335821 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.335868 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.336067 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.336133 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 22:59:06.336275 14128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.332657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:44995 (local address 127.29.78.129:59661)
0810 22:59:06.333048 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.333062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:06.333188 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:06.333438 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:06.333440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:06.333456 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:06.333623 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.333628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.334350 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.334354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.335212 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.335221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.335816 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.335819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.335907 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.335970 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 22:59:06.336049 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:59:06.336116 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":100,"spinlock_wait_cycles":14720,"thread_start_us":71,"threads_started":1}
I20260810 22:59:06.336370 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.332664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:41609 (local address 127.29.78.130:46781)
0810 22:59:06.332817 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.332822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.333093 (+ 271us) server_negotiation.cc:407] Connection header received
0810 22:59:06.333287 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.333290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.333355 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.333445 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:06.333753 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.334271 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.335355 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.335730 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.335861 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.335914 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.336156 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.336249 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:59:06.336824 14127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.332570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46781 (local address 127.29.78.129:41609)
0810 22:59:06.332995 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.333009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:06.333159 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:06.333454 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:06.333457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:06.333475 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:06.333635 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.333641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.334387 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.334393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.335218 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:06.335228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.335861 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.335865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:06.335956 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.336030 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 22:59:06.336553 (+ 523us) client_negotiation.cc:240] Negotiation successful
0810 22:59:06.336627 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":52,"thread_start_us":120,"threads_started":1}
I20260810 22:59:06.336854 14055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc55b83098954754a4b47eac960517a5" candidate_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a9c1232cf947c899b0bfe40397cc97" is_pre_election: true
I20260810 22:59:06.337177 14055 raft_consensus.cc:2454] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9056f9a7aeaf4a08ae21373be93e85eb in term 0.
I20260810 22:59:06.337510 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc55b83098954754a4b47eac960517a5" candidate_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706fe118d1f45e799b92cfe47c1a268" is_pre_election: true
I20260810 22:59:06.337730 13688 leader_election.cc:302] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05a9c1232cf947c899b0bfe40397cc97, 9056f9a7aeaf4a08ae21373be93e85eb; no voters:
I20260810 22:59:06.337852 13906 raft_consensus.cc:2454] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9056f9a7aeaf4a08ae21373be93e85eb in term 0.
I20260810 22:59:06.338037 13812 raft_consensus.cc:2781] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:59:06.338171 13812 raft_consensus.cc:482] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:59:06.338224 13812 raft_consensus.cc:3037] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:06.339392 13812 raft_consensus.cc:504] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.339772 13812 leader_election.cc:288] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 1 election: Requested vote from peers b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781), 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
I20260810 22:59:06.340132 14055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc55b83098954754a4b47eac960517a5" candidate_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05a9c1232cf947c899b0bfe40397cc97"
I20260810 22:59:06.340138 13906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc55b83098954754a4b47eac960517a5" candidate_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b706fe118d1f45e799b92cfe47c1a268"
I20260810 22:59:06.340258 13906 raft_consensus.cc:3037] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:06.340260 14055 raft_consensus.cc:3037] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:06.341673 14055 raft_consensus.cc:2454] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9056f9a7aeaf4a08ae21373be93e85eb in term 1.
I20260810 22:59:06.341673 13906 raft_consensus.cc:2454] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9056f9a7aeaf4a08ae21373be93e85eb in term 1.
I20260810 22:59:06.342000 13686 leader_election.cc:302] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05a9c1232cf947c899b0bfe40397cc97, 9056f9a7aeaf4a08ae21373be93e85eb, b706fe118d1f45e799b92cfe47c1a268; no voters:
I20260810 22:59:06.342202 13812 raft_consensus.cc:2781] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:06.342439 13812 raft_consensus.cc:686] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 LEADER]: Becoming Leader. State: Replica: 9056f9a7aeaf4a08ae21373be93e85eb, State: Running, Role: LEADER
I20260810 22:59:06.342749 13812 consensus_queue.cc:237] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.345999 13605 catalog_manager.cc:5495] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129). New cstate: current_term: 1 leader_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } health_report { overall_health: HEALTHY } } }
W20260810 22:59:06.361212 13957 tablet.cc:2369] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:06.387434 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.383788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57320 (local address 127.29.78.129:42985)
0810 22:59:06.384073 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:06.384085 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:06.384096 (+ 11us) server_negotiation.cc:407] Connection header received
0810 22:59:06.384160 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:06.384162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:06.384211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:06.384332 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:59:06.384597 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.385089 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.385987 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:06.386237 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:06.386366 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:06.386621 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:59:06.387010 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:59:06.387129 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:06.387193 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 22:59:06.387248 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260810 22:59:06.391259 14055 raft_consensus.cc:1261] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 FOLLOWER]: Refusing update from remote peer 9056f9a7aeaf4a08ae21373be93e85eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:06.391258 13906 raft_consensus.cc:1261] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Refusing update from remote peer 9056f9a7aeaf4a08ae21373be93e85eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:06.392017 14129 consensus_queue.cc:1035] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:06.392221 13812 consensus_queue.cc:1035] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:06.397965 14135 mvcc.cc:205] Tried to move back new op lower bound from 7317105649212383232 to 7317105649022365696. Current Snapshot: MvccSnapshot[applied={T|T < 7317105649212383232}]
I20260810 22:59:06.399399 14136 mvcc.cc:205] Tried to move back new op lower bound from 7317105649212383232 to 7317105649022365696. Current Snapshot: MvccSnapshot[applied={T|T < 7317105649212383232}]
I20260810 22:59:06.401176 14137 mvcc.cc:205] Tried to move back new op lower bound from 7317105649212383232 to 7317105649022365696. Current Snapshot: MvccSnapshot[applied={T|T < 7317105649212383232}]
I20260810 22:59:06.408751 13605 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:06.408947 13605 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:06.411386 13605 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Sending DeleteTablet for 3 replicas of tablet dc55b83098954754a4b47eac960517a5
I20260810 22:59:06.412134 14034 tablet_service.cc:1514] Processing DeleteTablet for tablet dc55b83098954754a4b47eac960517a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:06 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57720
I20260810 22:59:06.412153 13885 tablet_service.cc:1514] Processing DeleteTablet for tablet dc55b83098954754a4b47eac960517a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:06 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:43268
I20260810 22:59:06.412228 13734 tablet_service.cc:1514] Processing DeleteTablet for tablet dc55b83098954754a4b47eac960517a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:06 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:57318
I20260810 22:59:06.412631 14146 tablet_replica.cc:330] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: stopping tablet replica
I20260810 22:59:06.412657 14144 tablet_replica.cc:330] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: stopping tablet replica
I20260810 22:59:06.412743 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 13645
I20260810 22:59:06.412969 14146 raft_consensus.cc:2229] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:06.413125 14144 raft_consensus.cc:2229] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:59:06.413255 14146 raft_consensus.cc:2258] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:06.414779 14145 tablet_replica.cc:330] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: stopping tablet replica
I20260810 22:59:06.415040 14145 raft_consensus.cc:2229] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:06.415323 14145 raft_consensus.cc:2258] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:06.416510 14145 ts_tablet_manager.cc:1905] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:06.417092 14146 ts_tablet_manager.cc:1905] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:06.418926 14145 ts_tablet_manager.cc:1918] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:06.419063 14145 log.cc:1196] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/dc55b83098954754a4b47eac960517a5
I20260810 22:59:06.419394 14145 ts_tablet_manager.cc:1939] T dc55b83098954754a4b47eac960517a5 P b706fe118d1f45e799b92cfe47c1a268: Deleting consensus metadata
I20260810 22:59:06.420413 14146 ts_tablet_manager.cc:1918] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:06.420521 14146 log.cc:1196] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/dc55b83098954754a4b47eac960517a5
I20260810 22:59:06.420496 13589 catalog_manager.cc:4841] TS b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781): tablet dc55b83098954754a4b47eac960517a5 (table test-table [id=27c7e491dd334df6a35bd5ba90caf723]) successfully deleted
I20260810 22:59:06.420831 14146 ts_tablet_manager.cc:1939] T dc55b83098954754a4b47eac960517a5 P 05a9c1232cf947c899b0bfe40397cc97: Deleting consensus metadata
I20260810 22:59:06.421995 13591 catalog_manager.cc:4841] TS 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995): tablet dc55b83098954754a4b47eac960517a5 (table test-table [id=27c7e491dd334df6a35bd5ba90caf723]) successfully deleted
W20260810 22:59:06.423851 13589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:42985 (error 108)
I20260810 22:59:06.423889 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 13810
I20260810 22:59:06.426605 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.426313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:42985 (local address 127.0.0.1:57332)
0810 22:59:06.426431 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.426506 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 22:59:06.426785 13589 catalog_manager.cc:4584] TS 9056f9a7aeaf4a08ae21373be93e85eb (127.29.78.129:42985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dc55b83098954754a4b47eac960517a5: Network error: Client connection negotiation failed: client connection to 127.29.78.129:42985: connect: Connection refused (error 111)
I20260810 22:59:06.431332 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 13960
I20260810 22:59:06.439350 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 13568
I20260810 22:59:06.447093 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38533
--webserver_interface=127.29.78.190
--webserver_port=45669
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:06.552866 14148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:06.553121 14148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:06.553160 14148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:06.556840 14148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:59:06.556896 14148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:06.556913 14148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:06.556928 14148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:59:06.556942 14148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:59:06.562369 14148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=45669
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:06.563376 14148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:06.564361 14148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:06.572981 14154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:06.573060 14153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:06.573050 14156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:06.573486 14148 server_base.cc:1034] running on GCE node
I20260810 22:59:06.574156 14148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:06.575304 14148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:06.576519 14148 hybrid_clock.cc:648] HybridClock initialized: now 1786402746576498 us; error 38 us; skew 500 ppm
Aug 10 22:59:06 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402746, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:06.579471 14148 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:59:06.580363 14148 webserver.cc:460] Webserver started at http://127.29.78.190:45669/ using document root <none> and password file <none>
I20260810 22:59:06.580888 14148 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:06.580945 14148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:06.584724 14148 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:06.586886 14164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:06.588156 14148 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:06.588297 14148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "ea9356702dbe4b21ba44b10a94f8dfb8"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cc5ab17e7955180780ae4100713d4db9"
server_key_iv: "8a544753c146507babdf8413611c1266"
server_key_version: "encryptionkey@0"
I20260810 22:59:06.588888 14148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:06.603614 14148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:06.607520 14148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:06.607730 14148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:06.615614 14148 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:38533
I20260810 22:59:06.615924 14219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:38533 every 8 connection(s)
I20260810 22:59:06.616734 14148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:59:06.621043 14220 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:59:06.622684 14220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Bootstrap starting.
I20260810 22:59:06.623479 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 14148
I20260810 22:59:06.623910 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:42985
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:38533
--webserver_port=46199
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:59:06.646675 14220 log.cc:824] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:06.660475 14220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:59:06.660880 14220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8: Bootstrap complete.
I20260810 22:59:06.666138 14220 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:06.666524 14220 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea9356702dbe4b21ba44b10a94f8dfb8, State: Initialized, Role: FOLLOWER
I20260810 22:59:06.666965 14220 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:06.667100 14220 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:59:06.667181 14220 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:59:06.667289 14220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 2 FOLLOWER]: Advancing to term 3
I20260810 22:59:06.669030 14220 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:06.669407 14220 leader_election.cc:302] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea9356702dbe4b21ba44b10a94f8dfb8; no voters:
I20260810 22:59:06.669759 14220 leader_election.cc:288] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 22:59:06.669909 14225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 22:59:06.670136 14225 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [term 3 LEADER]: Becoming Leader. State: Replica: ea9356702dbe4b21ba44b10a94f8dfb8, State: Running, Role: LEADER
I20260810 22:59:06.670420 14225 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } }
I20260810 22:59:06.670842 14220 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:06.671278 14225 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea9356702dbe4b21ba44b10a94f8dfb8. Latest consensus state: current_term: 3 leader_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } } }
I20260810 22:59:06.671413 14225 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:06.671845 14230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:59:06.671955 14227 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea9356702dbe4b21ba44b10a94f8dfb8" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38533 } } }
I20260810 22:59:06.672140 14227 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea9356702dbe4b21ba44b10a94f8dfb8 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:06.676589 14230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:59:06.677140 14230 catalog_manager.cc:1259] Loaded cluster ID: b2e33eda3bfb4fe283611b6b58d46ec0
I20260810 22:59:06.677196 14230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 22:59:06.678200 14230 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f47ffe449ad google::LogMessage::Fail() at ??:0
@ 0x7f47ffe4788a google::LogMessage::SendToLog() at ??:0
@ 0x7f47ffe444de google::LogMessage::Flush() at ??:0
@ 0x7f47ffe45a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f480a783fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f480a784d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f480a781579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f480a7c5640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f48020963d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4802096c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f48020983f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4802087512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f48036cb6db start_thread at ??:0
@ 0x7f47fe94261f clone at ??:0
W20260810 22:59:06.765367 14223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:06.765676 14223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:06.765769 14223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:06.769743 14223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:06.769850 14223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:06.769963 14223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:59:06.775296 14223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:42985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=46199
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:06.776397 14223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:06.777442 14223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:06.787055 14269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:06.787058 14267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:06.787184 14266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:06.787863 14223 server_base.cc:1034] running on GCE node
I20260810 22:59:06.788257 14223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:06.788887 14223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:06.790072 14223 hybrid_clock.cc:648] HybridClock initialized: now 1786402746790048 us; error 41 us; skew 500 ppm
Aug 10 22:59:06 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402746, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:06.792964 14223 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:59:06.793865 14223 webserver.cc:460] Webserver started at http://127.29.78.129:46199/ using document root <none> and password file <none>
I20260810 22:59:06.794396 14223 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:06.794488 14223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:06.798252 14223 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 22:59:06.800554 14278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:06.802017 14223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 22:59:06.802145 14223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "9056f9a7aeaf4a08ae21373be93e85eb"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "15c7adefe4034bb3eeb229e07b78b79d"
server_key_iv: "0a4d9bfcd7f4c13553e4c7d14310e832"
server_key_version: "encryptionkey@0"
I20260810 22:59:06.802626 14223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:06.817479 14223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:06.820611 14223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:06.820815 14223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:06.821419 14223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:06.822638 14286 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:59:06.826581 14223 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:59:06.826658 14223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260810 22:59:06.826701 14223 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:59:06.828325 14223 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:59:06.828378 14223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:59:06.828645 14286 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap starting.
I20260810 22:59:06.838754 14223 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:42985
I20260810 22:59:06.839036 14401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:42985 every 8 connection(s)
I20260810 22:59:06.840045 14223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:59:06.842204 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 14223
I20260810 22:59:06.842803 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:46781
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:38533
--webserver_port=44835
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:59:06.849555 14286 log.cc:824] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:06.856463 14286 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:59:06.856832 14286 tablet_bootstrap.cc:492] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Bootstrap complete.
I20260810 22:59:06.858024 14286 ts_tablet_manager.cc:1397] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.007s
I20260810 22:59:06.860293 14286 raft_consensus.cc:348] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.860689 14286 raft_consensus.cc:729] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9056f9a7aeaf4a08ae21373be93e85eb, State: Initialized, Role: FOLLOWER
I20260810 22:59:06.861147 14286 consensus_queue.cc:260] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:06.862243 14286 ts_tablet_manager.cc:1428] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260810 22:59:06.931394 14404 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 22:59:06.842547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:42735)
0810 22:59:06.843804 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.843867 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:06.845023 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:06.930419 (+ 85396us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":135,"threads_started":1}
W20260810 22:59:06.932011 14402 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:38533 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:38533: Client connection negotiation failed: client connection to 127.29.78.190:38533: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 22:59:06.932766 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.932468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:60695)
0810 22:59:06.932609 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.932685 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 22:59:06.933408 14404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:06.933209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:59803)
0810 22:59:06.933311 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:06.933356 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 22:59:06.933713 14402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:59:06.981248 14405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:06.981478 14405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:06.981518 14405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:06.985162 14405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:06.985217 14405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:06.985319 14405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:59:06.990065 14405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:46781
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=44835
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:06.990948 14405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:06.991888 14405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:07.000964 14412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:07.001106 14405 server_base.cc:1034] running on GCE node
W20260810 22:59:07.000964 14413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:07.001025 14415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:07.001660 14405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:07.002354 14405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:07.003522 14405 hybrid_clock.cc:648] HybridClock initialized: now 1786402747003504 us; error 31 us; skew 500 ppm
Aug 10 22:59:07 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402747, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:07.006318 14405 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:59:07.007228 14405 webserver.cc:460] Webserver started at http://127.29.78.130:44835/ using document root <none> and password file <none>
I20260810 22:59:07.007800 14405 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:07.007870 14405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:07.011693 14405 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 22:59:07.013700 14423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:07.014937 14405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:07.015079 14405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "b706fe118d1f45e799b92cfe47c1a268"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "0c132b4ccacaad88fb6b248e9d75d249"
server_key_iv: "047c15f6e4574f8cf7ecc9fcbd00df49"
server_key_version: "encryptionkey@0"
I20260810 22:59:07.015643 14405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:07.047056 14405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:07.050028 14405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:07.050258 14405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:07.050864 14405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:07.051760 14405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:07.051841 14405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:07.051913 14405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:07.051959 14405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:07.061697 14405 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:46781
I20260810 22:59:07.061721 14548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:46781 every 8 connection(s)
I20260810 22:59:07.062769 14405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:59:07.065093 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 14405
I20260810 22:59:07.065616 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:44995
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:38533
--webserver_port=33775
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:59:07.066792 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:07.064923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:40133)
0810 22:59:07.065693 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:07.065884 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":120,"threads_started":1}
W20260810 22:59:07.067240 14549 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:38533 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:38533: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
I20260810 22:59:07.067895 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:07.067611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:56793)
0810 22:59:07.067754 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:07.067816 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 22:59:07.068657 14551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:07.068408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:45311)
0810 22:59:07.068559 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:07.068604 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 22:59:07.068910 14549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 22:59:07.173002 14552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:07.173233 14552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:07.173290 14552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:07.176918 14552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:07.176970 14552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:07.177019 14552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:59:07.182191 14552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:44995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=33775
--tserver_master_addrs=127.29.78.190:38533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:07.183171 14552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:07.184147 14552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:07.193347 14560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:07.193434 14552 server_base.cc:1034] running on GCE node
W20260810 22:59:07.193341 14558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:07.193341 14557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:07.194049 14552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:07.194684 14552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:07.195859 14552 hybrid_clock.cc:648] HybridClock initialized: now 1786402747195850 us; error 36 us; skew 500 ppm
Aug 10 22:59:07 dist-test-slave-6nmv krb5kdc[12971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402747, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:07.198711 14552 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:59:07.199563 14552 webserver.cc:460] Webserver started at http://127.29.78.131:33775/ using document root <none> and password file <none>
I20260810 22:59:07.200122 14552 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:07.200214 14552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:07.204244 14552 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 22:59:07.206362 14567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:07.207731 14552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:07.207859 14552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "05a9c1232cf947c899b0bfe40397cc97"
format_stamp: "Formatted at 2026-08-10 22:59:04 on dist-test-slave-6nmv"
server_key: "cca2474ead123d3f2d1c910d5857fcba"
server_key_iv: "4705902cc21aefbcb9a68d14b2a5aaff"
server_key_version: "encryptionkey@0"
I20260810 22:59:07.208389 14552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:07.223130 14552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:07.226230 14552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:07.226421 14552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:07.226977 14552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:07.227780 14552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:07.227854 14552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:07.227924 14552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:07.227973 14552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:07.236775 14552 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:44995
I20260810 22:59:07.236846 14694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:44995 every 8 connection(s)
I20260810 22:59:07.237780 14552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:59:07.241639 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:07.240064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:54809)
0810 22:59:07.240521 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:07.240934 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":119,"threads_started":1}
W20260810 22:59:07.242169 14695 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:38533 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:38533: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
I20260810 22:59:07.242803 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:07.242564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:53061)
0810 22:59:07.242675 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:07.242725 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 22:59:07.243115 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 14552
I20260810 22:59:07.243494 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:07.243288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:47071)
0810 22:59:07.243385 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:07.243425 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 22:59:07.243680 14695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 22:59:07.934772 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:07.934328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:56867)
0810 22:59:07.934599 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:07.934654 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260810 22:59:08.070009 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:08.069533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:51193)
0810 22:59:08.069772 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:08.069893 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260810 22:59:08.244658 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:08.244244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:39147)
0810 22:59:08.244500 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:08.244556 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
I20260810 22:59:08.614482 14704 raft_consensus.cc:482] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:08.614763 14704 raft_consensus.cc:504] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:08.615917 14704 leader_election.cc:288] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781), 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
Aug 10 22:59:08 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402746, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.131@KRBTEST.COM
Aug 10 22:59:08 dist-test-slave-6nmv krb5kdc[12971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402746, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.130@KRBTEST.COM
I20260810 22:59:08.629585 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:08.616161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:44995 (local address 127.29.78.129:36735)
0810 22:59:08.616475 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:08.616490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:08.616588 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:08.618221 (+ 1633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:08.618227 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:08.618798 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:08.619398 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:08.619421 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.620831 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.620834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:08.621609 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:08.621628 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.622378 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.622382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:08.622562 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:08.623231 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:08.623256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:08.624861 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:08.627176 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:08.627181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:08.627193 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:08.627503 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:08.627839 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:08.627845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:08.627847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:08.627985 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:08.628562 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:08.628572 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:08.628825 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 22:59:08.629017 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 22:59:08.629353 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":46,"spinlock_wait_cycles":21120,"thread_start_us":53,"threads_started":1}
I20260810 22:59:08.629593 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:08.616266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:52063 (local address 127.29.78.130:46781)
0810 22:59:08.616793 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:08.616806 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:08.616840 (+ 34us) server_negotiation.cc:407] Connection header received
0810 22:59:08.617479 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:08.617485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:08.617871 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:08.618132 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:08.619616 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.620690 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.622367 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.622826 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.623003 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:08.625415 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:08.625419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:08.625429 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:08.625456 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:08.627175 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:08.627649 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:08.627653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:08.627657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:08.627759 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:08.628106 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:08.628108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:08.628109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:08.628442 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:08.628581 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:08.629153 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 22:59:08.629417 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":59,"threads_started":1}
I20260810 22:59:08.629627 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:08.616113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:46781 (local address 127.29.78.129:52063)
0810 22:59:08.616339 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:08.616355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:08.616463 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:08.618496 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:08.618498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:08.618836 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:08.619402 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:08.619420 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.621728 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.621730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:08.622257 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:08.622264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.622954 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.622958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:08.623035 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:08.623492 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:08.623504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:08.625237 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:08.627284 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:08.627286 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:08.627287 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:08.627525 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:08.627881 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:08.627884 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:08.627886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:08.627996 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:08.628561 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:08.628566 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:08.628821 (+ 255us) client_negotiation.cc:769] Sending connection context
0810 22:59:08.629006 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 22:59:08.629452 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":203,"spinlock_wait_cycles":39040,"thread_start_us":100,"threads_started":1}
I20260810 22:59:08.629626 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:08.616552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:36735 (local address 127.29.78.131:44995)
0810 22:59:08.616858 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:08.616877 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:08.616906 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:59:08.617490 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:08.617495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:08.617870 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:08.618151 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:08.619617 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.620694 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.621719 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:08.622251 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:08.622423 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:08.625113 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:08.625119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:08.625136 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:08.625175 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:08.627063 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:08.627610 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:08.627616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:08.627622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:08.627735 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:08.628079 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:08.628082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:08.628084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:08.628448 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:08.628573 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:08.629150 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 22:59:08.629431 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":85,"threads_started":1}
W20260810 22:59:08.632747 14281 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
W20260810 22:59:08.632998 14283 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
I20260810 22:59:08.633085 14283 leader_election.cc:302] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9056f9a7aeaf4a08ae21373be93e85eb; no voters: 05a9c1232cf947c899b0bfe40397cc97, b706fe118d1f45e799b92cfe47c1a268
I20260810 22:59:08.633358 14704 raft_consensus.cc:2726] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:59:08.935743 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:08.935475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:50371)
0810 22:59:08.935600 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:08.935658 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 22:59:09.071309 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:09.070810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:41555)
0810 22:59:09.071093 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:09.071164 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260810 22:59:09.245702 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:09.245282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:51639)
0810 22:59:09.245555 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:09.245612 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260810 22:59:09.936858 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:09.936460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:54859)
0810 22:59:09.936707 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:09.936758 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20260810 22:59:10.072654 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:10.072130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:37817)
0810 22:59:10.072493 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:10.072554 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":104,"threads_started":1}
I20260810 22:59:10.246863 14713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:10.246440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:34543)
0810 22:59:10.246710 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:10.246759 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":104,"threads_started":1}
I20260810 22:59:10.389708 14714 raft_consensus.cc:482] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:10.389894 14714 raft_consensus.cc:504] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:10.390398 14714 leader_election.cc:288] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781), 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
W20260810 22:59:10.391211 14283 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
W20260810 22:59:10.391367 14281 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
I20260810 22:59:10.391451 14281 leader_election.cc:302] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9056f9a7aeaf4a08ae21373be93e85eb; no voters: 05a9c1232cf947c899b0bfe40397cc97, b706fe118d1f45e799b92cfe47c1a268
I20260810 22:59:10.391691 14714 raft_consensus.cc:2726] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:59:10.938099 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:10.937605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:50385)
0810 22:59:10.937926 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:10.937984 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260810 22:59:11.073977 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:11.073465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:46959)
0810 22:59:11.073728 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:11.073786 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260810 22:59:11.248011 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:11.247621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:48285)
0810 22:59:11.247880 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:11.247927 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260810 22:59:11.939496 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:11.939012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:49351)
0810 22:59:11.939340 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:11.939397 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":76,"threads_started":1}
I20260810 22:59:11.964183 14719 raft_consensus.cc:482] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:11.964329 14719 raft_consensus.cc:504] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:11.964833 14719 leader_election.cc:288] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781), 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
W20260810 22:59:11.965713 14283 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
W20260810 22:59:11.965802 14281 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
I20260810 22:59:11.965950 14281 leader_election.cc:302] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9056f9a7aeaf4a08ae21373be93e85eb; no voters: 05a9c1232cf947c899b0bfe40397cc97, b706fe118d1f45e799b92cfe47c1a268
I20260810 22:59:11.966135 14719 raft_consensus.cc:2726] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:59:12.075170 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:12.074750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:46625)
0810 22:59:12.075020 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:12.075074 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":105,"threads_started":1}
I20260810 22:59:12.249130 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:12.248704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:58185)
0810 22:59:12.248951 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:12.249007 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":88,"threads_started":1}
I20260810 22:59:12.940733 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:12.940302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:51821)
0810 22:59:12.940581 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:12.940636 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260810 22:59:13.076354 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:13.075971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:57421)
0810 22:59:13.076208 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:13.076260 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":75,"threads_started":1}
I20260810 22:59:13.250324 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:13.249899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:48173)
0810 22:59:13.250178 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:13.250231 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260810 22:59:13.942039 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:13.941541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:53615)
0810 22:59:13.941859 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:13.941940 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
I20260810 22:59:14.077569 14728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:14.077107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:57227)
0810 22:59:14.077422 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:14.077475 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":84,"threads_started":1}
I20260810 22:59:14.251487 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:14.251063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:43091)
0810 22:59:14.251339 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:14.251390 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":74,"threads_started":1}
I20260810 22:59:14.943558 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:14.943074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:37667)
0810 22:59:14.943383 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:14.943450 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":161,"threads_started":1}
I20260810 22:59:14.993505 14731 raft_consensus.cc:482] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:14.993726 14731 raft_consensus.cc:504] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:14.994279 14731 leader_election.cc:288] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781), 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
W20260810 22:59:14.995018 14281 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
W20260810 22:59:14.995141 14283 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
I20260810 22:59:14.995260 14283 leader_election.cc:302] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9056f9a7aeaf4a08ae21373be93e85eb; no voters: 05a9c1232cf947c899b0bfe40397cc97, b706fe118d1f45e799b92cfe47c1a268
I20260810 22:59:14.995421 14731 raft_consensus.cc:2726] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:59:15.078832 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:15.078337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:56785)
0810 22:59:15.078684 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:15.078741 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":76,"threads_started":1}
I20260810 22:59:15.252765 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:15.252345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:58747)
0810 22:59:15.252606 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:15.252658 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":96,"threads_started":1}
I20260810 22:59:15.945000 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:15.944563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:45691)
0810 22:59:15.944838 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:15.944903 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":84,"threads_started":1}
I20260810 22:59:16.079984 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:16.079577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:34777)
0810 22:59:16.079830 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:16.079887 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260810 22:59:16.253937 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:16.253476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:36719)
0810 22:59:16.253728 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:16.253779 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":104,"threads_started":1}
I20260810 22:59:16.946534 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:16.946061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:48847)
0810 22:59:16.946340 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:16.946423 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260810 22:59:17.081202 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:17.080795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:35985)
0810 22:59:17.081052 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:17.081104 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":94,"threads_started":1}
I20260810 22:59:17.255179 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:17.254771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:38275)
0810 22:59:17.255035 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:17.255085 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
I20260810 22:59:17.947748 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:17.947290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:43033)
0810 22:59:17.947586 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:17.947647 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":80,"threads_started":1}
I20260810 22:59:18.082536 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:18.082040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:33093)
0810 22:59:18.082334 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:18.082397 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":107,"threads_started":1}
I20260810 22:59:18.256373 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:18.255993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:35989)
0810 22:59:18.256232 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:18.256282 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
I20260810 22:59:18.949076 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:18.948623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:58317)
0810 22:59:18.948924 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:18.948979 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":78,"threads_started":1}
I20260810 22:59:19.083629 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:19.083247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:46143)
0810 22:59:19.083480 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:19.083535 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
I20260810 22:59:19.257452 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:19.257080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:48405)
0810 22:59:19.257314 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:19.257366 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":82,"threads_started":1}
I20260810 22:59:19.939261 14748 raft_consensus.cc:482] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:19.939391 14748 raft_consensus.cc:504] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b706fe118d1f45e799b92cfe47c1a268" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 46781 } } peers { permanent_uuid: "05a9c1232cf947c899b0bfe40397cc97" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 44995 } } peers { permanent_uuid: "9056f9a7aeaf4a08ae21373be93e85eb" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 42985 } }
I20260810 22:59:19.939762 14748 leader_election.cc:288] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781), 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995)
W20260810 22:59:19.940548 14283 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 05a9c1232cf947c899b0bfe40397cc97 (127.29.78.131:44995): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
W20260810 22:59:19.940692 14281 leader_election.cc:341] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b706fe118d1f45e799b92cfe47c1a268 (127.29.78.130:46781): Not found: Tablet not found: dc55b83098954754a4b47eac960517a5
I20260810 22:59:19.940799 14281 leader_election.cc:302] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9056f9a7aeaf4a08ae21373be93e85eb; no voters: 05a9c1232cf947c899b0bfe40397cc97, b706fe118d1f45e799b92cfe47c1a268
I20260810 22:59:19.940995 14748 raft_consensus.cc:2726] T dc55b83098954754a4b47eac960517a5 P 9056f9a7aeaf4a08ae21373be93e85eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 22:59:19.950131 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:19.949737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:54729)
0810 22:59:19.950001 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:19.950049 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
I20260810 22:59:20.084826 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:20.084423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:38133)
0810 22:59:20.084675 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:20.084731 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260810 22:59:20.258637 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:20.258228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:47275)
0810 22:59:20.258483 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:20.258541 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260810 22:59:20.951396 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:20.950972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:34257)
0810 22:59:20.951238 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:20.951289 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260810 22:59:21.086095 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:21.085646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:41785)
0810 22:59:21.085931 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:21.085998 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260810 22:59:21.259862 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:21.259463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:49997)
0810 22:59:21.259711 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:21.259763 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260810 22:59:21.952916 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:21.952420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.129:37593)
0810 22:59:21.952718 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:21.952799 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":84,"threads_started":1}
I20260810 22:59:22.087329 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:22.086907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.130:60759)
0810 22:59:22.087174 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:22.087237 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260810 22:59:22.253298 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 14223
I20260810 22:59:22.261336 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:22.260907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38533 (local address 127.29.78.131:33395)
0810 22:59:22.261186 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:22.261242 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260810 22:59:22.263013 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 14405
I20260810 22:59:22.270324 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 14552
2026-08-10T22:59:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19431 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 22:59:22 dist-test-slave-6nmv krb5kdc[14760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 22:59:22 dist-test-slave-6nmv krb5kdc[14760](info): set up 2 sockets
Aug 10 22:59:22 dist-test-slave-6nmv krb5kdc[14760](info): commencing operation
krb5kdc: starting...
W20260810 22:59:23.320007 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 22:59:23 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T22:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T22:59:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 22:59:23.436749 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35109
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:23.544939 14774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:23.545217 14774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:23.545281 14774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:23.549093 14774 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:59:23.549177 14774 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:59:23.549223 14774 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:59:23.549274 14774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:59:23.549321 14774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:23.549386 14774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:23.549424 14774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:59:23.549461 14774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:59:23.554637 14774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:23.555727 14774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:23.556778 14774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:23.565770 14779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:23.565776 14782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:23.565776 14780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:23.565848 14774 server_base.cc:1034] running on GCE node
I20260810 22:59:23.566653 14774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:23.567773 14774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:23.568934 14774 hybrid_clock.cc:648] HybridClock initialized: now 1786402763568917 us; error 30 us; skew 500 ppm
Aug 10 22:59:23 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402763, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:23.571842 14774 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:59:23.572808 14774 webserver.cc:460] Webserver started at http://127.29.78.190:38247/ using document root <none> and password file <none>
I20260810 22:59:23.573396 14774 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:23.573459 14774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:23.573724 14774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:23.575793 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "635af0e94ecd4e279bd10ea03368dce3"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "41a25033c4e9d1591476a24dc3a2f172"
server_key_iv: "1b95dc53aa6942aaf9cd3cb59b912b78"
server_key_version: "encryptionkey@0"
I20260810 22:59:23.576380 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "635af0e94ecd4e279bd10ea03368dce3"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "41a25033c4e9d1591476a24dc3a2f172"
server_key_iv: "1b95dc53aa6942aaf9cd3cb59b912b78"
server_key_version: "encryptionkey@0"
I20260810 22:59:23.580088 14774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 22:59:23.582659 14789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:23.583853 14774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:23.583990 14774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "635af0e94ecd4e279bd10ea03368dce3"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "41a25033c4e9d1591476a24dc3a2f172"
server_key_iv: "1b95dc53aa6942aaf9cd3cb59b912b78"
server_key_version: "encryptionkey@0"
I20260810 22:59:23.584093 14774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:23.605394 14774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:23.608407 14774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:23.608618 14774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:23.616057 14774 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35109
I20260810 22:59:23.616349 14846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35109 every 8 connection(s)
I20260810 22:59:23.617004 14774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:59:23.619668 14847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:23.623078 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 14774
I20260810 22:59:23.623205 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 22:59:23.623479 30010 external_mini_cluster.cc:1428] Setting key 6b887a19eec3fb733e5c8867e988db58
I20260810 22:59:23.625422 14847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Bootstrap starting.
I20260810 22:59:23.627887 14847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:23.628638 14847 log.cc:824] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:23.630874 14847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: No bootstrap required, opened a new log
Aug 10 22:59:23 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:23.635192 14847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:23.635468 14847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:23.635586 14847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635af0e94ecd4e279bd10ea03368dce3, State: Initialized, Role: FOLLOWER
I20260810 22:59:23.636085 14847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:23.636296 14847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:59:23.636363 14847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:59:23.636451 14847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:23.637553 14847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:23.637912 14847 leader_election.cc:302] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635af0e94ecd4e279bd10ea03368dce3; no voters:
I20260810 22:59:23.638258 14847 leader_election.cc:288] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 22:59:23.638379 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:23.624885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.29.78.190:35109)
0810 22:59:23.625278 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:23.625293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:23.625326 (+ 33us) server_negotiation.cc:407] Connection header received
0810 22:59:23.626030 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:23.626047 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:23.626344 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:23.626597 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:23.627474 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:23.628518 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:23.629557 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:23.630102 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:23.630224 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:23.633195 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:23.633200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:23.633212 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:23.633238 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:23.635095 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:23.635579 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:23.635585 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:23.635590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:23.635737 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:23.636061 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:23.636064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:23.636066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:23.636650 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:23.636832 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:23.637081 (+ 249us) server_negotiation.cc:299] Negotiation successful
0810 22:59:23.637380 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":137,"threads_started":1}
I20260810 22:59:23.638384 14852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:23.639520 14852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 LEADER]: Becoming Leader. State: Replica: 635af0e94ecd4e279bd10ea03368dce3, State: Running, Role: LEADER
I20260810 22:59:23.639564 14847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:23.642009 14852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:23.643519 14854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635af0e94ecd4e279bd10ea03368dce3. Latest consensus state: current_term: 1 leader_uuid: "635af0e94ecd4e279bd10ea03368dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } } }
I20260810 22:59:23.643725 14853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635af0e94ecd4e279bd10ea03368dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } } }
I20260810 22:59:23.644204 14853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:23.644186 14854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: This master's current role is: LEADER
W20260810 22:59:23.647204 14867 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 22:59:23.647292 14867 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 22:59:23.647368 14870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:59:23.648092 14870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:59:23.653772 14870 catalog_manager.cc:1347] Generated new cluster ID: 6736d0136f5e435f84ea787355a16659
I20260810 22:59:23.653920 14870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:59:23.674665 14870 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 22:59:23.675661 14870 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:59:23.688313 14870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Generated new TSK 0
I20260810 22:59:23.689129 14870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 22:59:23.729558 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35109
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:23.853677 14875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:23.853909 14875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:23.853945 14875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:23.857538 14875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:23.857582 14875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:23.857632 14875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:59:23.862092 14875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:23.862917 14875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:23.863821 14875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:23.872604 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:23.872850 14875 server_base.cc:1034] running on GCE node
W20260810 22:59:23.872620 14885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:23.872605 14883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:23.873344 14875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:23.873944 14875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:23.875101 14875 hybrid_clock.cc:648] HybridClock initialized: now 1786402763875079 us; error 52 us; skew 500 ppm
Aug 10 22:59:23 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402763, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:23.877642 14875 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:59:23.878487 14875 webserver.cc:460] Webserver started at http://127.29.78.129:37295/ using document root <none> and password file <none>
I20260810 22:59:23.878969 14875 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:23.879025 14875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:23.879196 14875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:23.881098 14875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "d1a0ae08b6da9de2523a4d9325af3e3b"
server_key_iv: "af98b882e14008f1abcc10b9691ceedb"
server_key_version: "encryptionkey@0"
I20260810 22:59:23.881623 14875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "d1a0ae08b6da9de2523a4d9325af3e3b"
server_key_iv: "af98b882e14008f1abcc10b9691ceedb"
server_key_version: "encryptionkey@0"
I20260810 22:59:23.885206 14875 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 22:59:23.887737 14894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:23.888945 14875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 22:59:23.889037 14875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "d1a0ae08b6da9de2523a4d9325af3e3b"
server_key_iv: "af98b882e14008f1abcc10b9691ceedb"
server_key_version: "encryptionkey@0"
I20260810 22:59:23.889170 14875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:23.916929 14875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:23.919906 14875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:23.920105 14875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:23.920732 14875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:23.921584 14875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:23.921628 14875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:23.921687 14875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:23.921718 14875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:23.930253 14875 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38925
I20260810 22:59:23.930285 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38925 every 8 connection(s)
I20260810 22:59:23.931155 14875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:59:23.937501 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 14875
I20260810 22:59:23.937671 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 22:59:23.938040 30010 external_mini_cluster.cc:1428] Setting key fb8a84229cf0b7c8781067b90f851411
Aug 10 22:59:23 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402763, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:23.945533 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:23.933110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:59437 (local address 127.29.78.190:35109)
0810 22:59:23.933259 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:23.933263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:23.934356 (+ 1093us) server_negotiation.cc:407] Connection header received
0810 22:59:23.935243 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:23.935246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:23.935305 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:23.935407 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:23.936556 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:23.937093 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:23.938435 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:23.938695 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:23.938804 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:23.941763 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:23.941766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:23.941768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:23.941793 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:23.943237 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:23.943852 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:23.943855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:23.943856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:23.943946 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:23.944338 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:23.944341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:23.944343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:23.944528 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:23.944611 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:23.945270 (+ 659us) server_negotiation.cc:299] Negotiation successful
0810 22:59:23.945409 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 22:59:23.946694 15018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:23.933524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:59437)
0810 22:59:23.934207 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:23.934247 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:23.935039 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:23.935589 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:23.935596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:23.935946 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:23.936389 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:23.936412 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:23.937222 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:23.937228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:23.938301 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:23.938311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:23.938844 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:23.938847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:23.939009 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:23.939610 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:23.939625 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:23.941594 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:23.943358 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:23.943366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:23.943383 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:23.943733 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:23.944050 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:23.944055 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:23.944059 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:23.944246 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:23.944635 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:23.944643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:23.944989 (+ 346us) client_negotiation.cc:769] Sending connection context
0810 22:59:23.945200 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 22:59:23.945564 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":132,"threads_started":1}
I20260810 22:59:23.948205 15016 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35109
I20260810 22:59:23.948489 15016 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:23.949043 15016 heartbeater.cc:511] Master 127.29.78.190:35109 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 22:59:23.950703 14806 ts_manager.cc:194] Registered new tserver with Master: 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129:38925)
I20260810 22:59:23.951889 14806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:59437
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 22:59:23.966990 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35109
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:24.073401 15021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:24.073683 15021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:24.073756 15021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:24.077879 15021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:24.077956 15021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:24.078050 15021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:59:24.083148 15021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:24.084187 15021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:24.085222 15021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:24.095283 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:24.095364 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:24.095366 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:24.095594 15021 server_base.cc:1034] running on GCE node
I20260810 22:59:24.095988 15021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:24.096601 15021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:24.097754 15021 hybrid_clock.cc:648] HybridClock initialized: now 1786402764097733 us; error 33 us; skew 500 ppm
Aug 10 22:59:24 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402764, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:24.100433 15021 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:59:24.101300 15021 webserver.cc:460] Webserver started at http://127.29.78.130:38903/ using document root <none> and password file <none>
I20260810 22:59:24.101919 15021 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:24.102025 15021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:24.102288 15021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:24.104245 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "b0e19f60a15646099fc5574df30a96a6"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "b85d96632ff4a15b0f4486de505d8d01"
server_key_iv: "59e24190d4712cd659134bba1dfdcf80"
server_key_version: "encryptionkey@0"
I20260810 22:59:24.104802 15021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "b0e19f60a15646099fc5574df30a96a6"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "b85d96632ff4a15b0f4486de505d8d01"
server_key_iv: "59e24190d4712cd659134bba1dfdcf80"
server_key_version: "encryptionkey@0"
I20260810 22:59:24.108415 15021 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 22:59:24.110745 15038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:24.111931 15021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:24.112068 15021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "b0e19f60a15646099fc5574df30a96a6"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "b85d96632ff4a15b0f4486de505d8d01"
server_key_iv: "59e24190d4712cd659134bba1dfdcf80"
server_key_version: "encryptionkey@0"
I20260810 22:59:24.112174 15021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:24.123114 15021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:24.125641 15021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:24.125910 15021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:24.126443 15021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:24.127221 15021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:24.127291 15021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:24.127353 15021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:24.127405 15021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:24.136317 15021 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44739
I20260810 22:59:24.136348 15162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44739 every 8 connection(s)
I20260810 22:59:24.137187 15021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:59:24.142995 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 15021
I20260810 22:59:24.143097 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 22:59:24.143366 30010 external_mini_cluster.cc:1428] Setting key 9277bc4905de8b71256eacf47a77a72b
Aug 10 22:59:24 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402764, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:24.152194 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.139080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:46281 (local address 127.29.78.190:35109)
0810 22:59:24.139220 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.139229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.140326 (+ 1097us) server_negotiation.cc:407] Connection header received
0810 22:59:24.141367 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.141371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.141452 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.141547 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:24.142844 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.143494 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.144729 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.145063 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.145165 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.147989 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:24.147994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:24.147999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:24.148031 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:24.149773 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:24.150388 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:24.150393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:24.150395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:24.150497 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:24.150836 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:24.150843 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:24.150846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:24.151145 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:24.151263 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.151821 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.151992 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 22:59:24.152912 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.139439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:46281)
0810 22:59:24.140158 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.140203 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.141080 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.141723 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.141734 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.142172 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:24.142665 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.142679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.143613 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.143617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.144622 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.144633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.145215 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.145219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.145412 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.146016 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:24.146030 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:24.147774 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:24.149980 (+ 2206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:24.149988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:24.150004 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:24.150277 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:24.150597 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:24.150599 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:24.150600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:24.150732 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:24.151264 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:24.151268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:24.151516 (+ 248us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.151709 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.151976 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":108,"threads_started":1}
I20260810 22:59:24.154209 15163 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35109
I20260810 22:59:24.154445 15163 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 22:59:24.154987 15163 heartbeater.cc:511] Master 127.29.78.190:35109 requested a full tablet report, sending...
I20260810 22:59:24.156091 14806 ts_manager.cc:194] Registered new tserver with Master: b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
I20260810 22:59:24.156724 14806 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:46281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 22:59:24.173588 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35109
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:24.287680 15168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:24.287997 15168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:24.288080 15168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:24.292466 15168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:24.292559 15168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:24.292677 15168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:59:24.298084 15168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:24.299063 15168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:24.300107 15168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:24.309547 15177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:24.309556 15175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:24.309535 15174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:24.309932 15168 server_base.cc:1034] running on GCE node
I20260810 22:59:24.310330 15168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:24.310954 15168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:24.312152 15168 hybrid_clock.cc:648] HybridClock initialized: now 1786402764312136 us; error 34 us; skew 500 ppm
Aug 10 22:59:24 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402764, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:24.315122 15168 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:59:24.316154 15168 webserver.cc:460] Webserver started at http://127.29.78.131:34827/ using document root <none> and password file <none>
I20260810 22:59:24.316709 15168 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:24.316829 15168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:24.317094 15168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 22:59:24.319110 15168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "5f34641c91474107b3b74dcbf7c4849f"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "73be00c5a3218c448864fed9a2cde697"
server_key_iv: "759569c0a094e4271e7dbdd5bdbaa33e"
server_key_version: "encryptionkey@0"
I20260810 22:59:24.319694 15168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "5f34641c91474107b3b74dcbf7c4849f"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "73be00c5a3218c448864fed9a2cde697"
server_key_iv: "759569c0a094e4271e7dbdd5bdbaa33e"
server_key_version: "encryptionkey@0"
I20260810 22:59:24.323617 15168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 22:59:24.325929 15184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:24.327069 15168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:24.327201 15168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "5f34641c91474107b3b74dcbf7c4849f"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "73be00c5a3218c448864fed9a2cde697"
server_key_iv: "759569c0a094e4271e7dbdd5bdbaa33e"
server_key_version: "encryptionkey@0"
I20260810 22:59:24.327302 15168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:24.352514 15168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:24.355480 15168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:24.355681 15168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:24.356262 15168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:24.357160 15168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:24.357244 15168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:24.357321 15168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:24.357365 15168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:24.366606 15168 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:33769
I20260810 22:59:24.366628 15306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:33769 every 8 connection(s)
I20260810 22:59:24.367442 15168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:59:24.370736 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 15168
I20260810 22:59:24.370834 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 22:59:24.371165 30010 external_mini_cluster.cc:1428] Setting key 59942aef890ba66ea24ed4f388e7ccbd
Aug 10 22:59:24 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402764, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:24.381300 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.369165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:34355 (local address 127.29.78.190:35109)
0810 22:59:24.369337 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.369342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.370431 (+ 1089us) server_negotiation.cc:407] Connection header received
0810 22:59:24.371510 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.371513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.371575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.371677 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:24.373079 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.373751 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.374881 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.375099 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.375176 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.378073 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:24.378077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:24.378079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:24.378112 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:24.379434 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:24.379934 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:24.379937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:24.379938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:24.380023 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:24.380332 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:24.380334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:24.380335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:24.380513 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:24.380589 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.381067 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.381177 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 22:59:24.382161 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.369554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:34355)
0810 22:59:24.370284 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.370324 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.371248 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.371899 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.371926 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.372372 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:24.372946 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.372960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.373905 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.373909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.374780 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.374789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.375223 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.375225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.375447 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.376048 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:24.376063 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:24.377875 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:24.379541 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:24.379546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:24.379557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:24.379813 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:24.380103 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:24.380105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:24.380107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:24.380244 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:24.380600 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:24.380604 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:24.380840 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.380999 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.381253 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":103,"threads_started":1}
I20260810 22:59:24.383353 15307 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35109
I20260810 22:59:24.383608 15307 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:24.384181 15307 heartbeater.cc:511] Master 127.29.78.190:35109 requested a full tablet report, sending...
I20260810 22:59:24.385318 14805 ts_manager.cc:194] Registered new tserver with Master: 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769)
I20260810 22:59:24.385998 14805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:34355
I20260810 22:59:24.396968 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:59:24.411995 14850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.398959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.29.78.190:35109)
0810 22:59:24.399094 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.399097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.399299 (+ 202us) server_negotiation.cc:407] Connection header received
0810 22:59:24.399403 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.399405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.399450 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.399519 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:24.400177 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.400698 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.401504 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.401701 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.401773 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.409166 (+ 7393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:24.409169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:24.409171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:24.409196 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:24.410528 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:24.410914 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:24.410916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:24.410918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:24.410987 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:24.411216 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:24.411218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:24.411219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:24.411431 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:24.411552 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.411782 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.411878 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 22:59:24.414914 14805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:59:24.418150 14805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:59:24.432724 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.426079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.29.78.130:44739)
0810 22:59:24.427744 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.427755 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.427774 (+ 19us) server_negotiation.cc:407] Connection header received
0810 22:59:24.427838 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.427842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.428005 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.428139 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:24.428603 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.429844 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.431236 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.432018 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.432263 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.432422 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.432486 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.432568 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1567,"thread_start_us":155,"threads_started":1}
I20260810 22:59:24.432734 15320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.426462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38925 (local address 127.0.0.1:38758)
0810 22:59:24.426859 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.426875 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.426965 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.428024 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.428033 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.428061 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:24.428225 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.428232 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.429719 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.429725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.431138 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.431150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.432319 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.432323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.432421 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.432448 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.432499 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.432569 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":20,"thread_start_us":99,"threads_started":1}
I20260810 22:59:24.432708 15318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.426013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44739 (local address 127.0.0.1:60176)
0810 22:59:24.426372 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.426421 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.426548 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.428308 (+ 1760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.428314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.428334 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:24.428459 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.428466 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.430014 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.430017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.431113 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.431123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.432138 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.432142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.432266 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.432313 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.432411 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.432529 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260810 22:59:24.433285 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.426876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38758 (local address 127.29.78.129:38925)
0810 22:59:24.427524 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.427539 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.427564 (+ 25us) server_negotiation.cc:407] Connection header received
0810 22:59:24.427649 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.427654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.427850 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.428381 (+ 531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:24.428566 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.429571 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.431289 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.432180 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.432822 (+ 642us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.432989 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.433040 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.433128 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":71,"threads_started":1}
I20260810 22:59:24.434041 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.426721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33769 (local address 127.0.0.1:49212)
0810 22:59:24.427162 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.427176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.427264 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.427968 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.428650 (+ 682us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.428689 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:24.428844 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.428853 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.430468 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.430471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.431718 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.431729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.433594 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.433598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.433688 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.433709 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.433797 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.433900 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260810 22:59:24.434171 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.427214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.29.78.131:33769)
0810 22:59:24.427586 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.427595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.427613 (+ 18us) server_negotiation.cc:407] Connection header received
0810 22:59:24.427672 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.427675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.427818 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.427988 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:24.428988 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.430315 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.431873 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.433486 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.433597 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.433724 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.433914 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.434017 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":187,"threads_started":1}
I20260810 22:59:24.435976 15091 tablet_service.cc:1467] Processing CreateTablet for tablet bf77561589184498932ac1b65319a5c2 (DEFAULT_TABLE table=test-table [id=f207e28fd593406088334beaf8138620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:24.436241 14944 tablet_service.cc:1467] Processing CreateTablet for tablet bf77561589184498932ac1b65319a5c2 (DEFAULT_TABLE table=test-table [id=f207e28fd593406088334beaf8138620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:24.437047 15091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf77561589184498932ac1b65319a5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:24.436661 15237 tablet_service.cc:1467] Processing CreateTablet for tablet bf77561589184498932ac1b65319a5c2 (DEFAULT_TABLE table=test-table [id=f207e28fd593406088334beaf8138620]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:24.437346 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf77561589184498932ac1b65319a5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:24.437577 15237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf77561589184498932ac1b65319a5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:24.443761 15324 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Bootstrap starting.
I20260810 22:59:24.445055 15325 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Bootstrap starting.
I20260810 22:59:24.446173 15326 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap starting.
I20260810 22:59:24.446448 15324 tablet_bootstrap.cc:654] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:24.447268 15325 tablet_bootstrap.cc:654] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:24.447333 15324 log.cc:824] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:24.448087 15325 log.cc:824] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:24.448167 15326 tablet_bootstrap.cc:654] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:24.448935 15326 log.cc:824] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:24.449740 15324 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: No bootstrap required, opened a new log
I20260810 22:59:24.449968 15324 ts_tablet_manager.cc:1397] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 22:59:24.450119 15325 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: No bootstrap required, opened a new log
I20260810 22:59:24.450342 15325 ts_tablet_manager.cc:1397] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260810 22:59:24.450887 15326 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: No bootstrap required, opened a new log
I20260810 22:59:24.451068 15326 ts_tablet_manager.cc:1397] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:59:24.453220 15325 raft_consensus.cc:348] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.453461 15325 raft_consensus.cc:374] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:24.453531 15325 raft_consensus.cc:729] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f34641c91474107b3b74dcbf7c4849f, State: Initialized, Role: FOLLOWER
I20260810 22:59:24.454030 15325 consensus_queue.cc:260] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.453986 15324 raft_consensus.cc:348] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.454284 15324 raft_consensus.cc:374] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:24.454381 15324 raft_consensus.cc:729] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e19f60a15646099fc5574df30a96a6, State: Initialized, Role: FOLLOWER
I20260810 22:59:24.454622 15307 heartbeater.cc:503] Master 127.29.78.190:35109 was elected leader, sending a full tablet report...
I20260810 22:59:24.454792 15324 consensus_queue.cc:260] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.454990 15325 ts_tablet_manager.cc:1428] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 22:59:24.455705 15324 ts_tablet_manager.cc:1428] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 22:59:24.455870 15326 raft_consensus.cc:348] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.456080 15163 heartbeater.cc:503] Master 127.29.78.190:35109 was elected leader, sending a full tablet report...
I20260810 22:59:24.456180 15326 raft_consensus.cc:374] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:24.456238 15326 raft_consensus.cc:729] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6d7a7d94814c9aba1b4753d7cd0f1c, State: Initialized, Role: FOLLOWER
I20260810 22:59:24.456676 15326 consensus_queue.cc:260] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.457309 15016 heartbeater.cc:503] Master 127.29.78.190:35109 was elected leader, sending a full tablet report...
I20260810 22:59:24.457620 15326 ts_tablet_manager.cc:1428] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260810 22:59:24.618415 15308 tablet.cc:2369] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:24.625065 15330 raft_consensus.cc:482] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:24.625285 15330 raft_consensus.cc:504] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.628700 15330 leader_election.cc:288] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739), 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129:38925)
I20260810 22:59:24.630959 15332 raft_consensus.cc:482] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:24.631280 15309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.627095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44739 (local address 127.29.78.131:38507)
0810 22:59:24.627299 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.627315 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.627419 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.627761 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.627764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.627784 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:24.627896 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.627901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.628879 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.628882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.629990 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.630001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.630790 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.630794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.630928 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.630955 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.631034 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.631120 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":1}
I20260810 22:59:24.631207 15332 raft_consensus.cc:504] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.631772 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.627175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38507 (local address 127.29.78.130:44739)
0810 22:59:24.627295 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.627300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.627365 (+ 65us) server_negotiation.cc:407] Connection header received
0810 22:59:24.627518 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.627521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.627567 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.627642 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:24.628022 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.628757 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.630185 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.630658 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.630790 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.630845 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.631564 (+ 719us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.631661 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:59:24.632222 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.627800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38925 (local address 127.29.78.131:38807)
0810 22:59:24.628109 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.628122 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.628210 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.628620 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.628622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.628637 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:24.628750 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.628756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.630046 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.630052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.630915 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.630925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.631832 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.631836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.631943 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.631972 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.632042 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.632118 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":124,"threads_started":1}
I20260810 22:59:24.632264 15322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.628266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38807 (local address 127.29.78.129:38925)
0810 22:59:24.628411 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.628416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.628429 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:59:24.628483 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.628485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.628534 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.628613 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:24.628878 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.629668 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.631057 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.631697 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.631832 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.631880 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.632065 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.632159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 22:59:24.632341 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf77561589184498932ac1b65319a5c2" candidate_uuid: "5f34641c91474107b3b74dcbf7c4849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e19f60a15646099fc5574df30a96a6" is_pre_election: true
I20260810 22:59:24.632678 15112 raft_consensus.cc:2454] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f34641c91474107b3b74dcbf7c4849f in term 0.
I20260810 22:59:24.632791 14964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf77561589184498932ac1b65319a5c2" candidate_uuid: "5f34641c91474107b3b74dcbf7c4849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" is_pre_election: true
I20260810 22:59:24.633038 14964 raft_consensus.cc:2454] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f34641c91474107b3b74dcbf7c4849f in term 0.
I20260810 22:59:24.633164 15189 leader_election.cc:302] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6; no voters:
I20260810 22:59:24.633435 15330 raft_consensus.cc:2781] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:59:24.633559 15330 raft_consensus.cc:482] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:59:24.633625 15330 raft_consensus.cc:3037] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:24.634681 15332 leader_election.cc:288] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739), 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769)
I20260810 22:59:24.635262 15330 raft_consensus.cc:504] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.635948 15330 leader_election.cc:288] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [CANDIDATE]: Term 1 election: Requested vote from peers b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739), 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129:38925)
I20260810 22:59:24.635933 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf77561589184498932ac1b65319a5c2" candidate_uuid: "5f34641c91474107b3b74dcbf7c4849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e19f60a15646099fc5574df30a96a6"
I20260810 22:59:24.636041 15112 raft_consensus.cc:3037] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:24.636427 14964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf77561589184498932ac1b65319a5c2" candidate_uuid: "5f34641c91474107b3b74dcbf7c4849f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c"
I20260810 22:59:24.636534 14964 raft_consensus.cc:3037] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:24.637367 15112 raft_consensus.cc:2454] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f34641c91474107b3b74dcbf7c4849f in term 1.
I20260810 22:59:24.637666 14964 raft_consensus.cc:2454] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f34641c91474107b3b74dcbf7c4849f in term 1.
I20260810 22:59:24.637687 15189 leader_election.cc:302] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6; no voters:
I20260810 22:59:24.637936 15330 raft_consensus.cc:2781] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:24.638170 15330 raft_consensus.cc:686] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 1 LEADER]: Becoming Leader. State: Replica: 5f34641c91474107b3b74dcbf7c4849f, State: Running, Role: LEADER
I20260810 22:59:24.638266 15319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.634139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:39581 (local address 127.29.78.130:44739)
0810 22:59:24.634248 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.634251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.634264 (+ 13us) server_negotiation.cc:407] Connection header received
0810 22:59:24.634302 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.634304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.634350 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.634428 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:24.634724 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.635457 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.637039 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.637545 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.637620 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.637664 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.638061 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.638148 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 22:59:24.638381 15164 tablet.cc:2369] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:24.638646 15335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.633725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44739 (local address 127.29.78.129:39581)
0810 22:59:24.634015 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.634029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.634117 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.634444 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.634450 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.634476 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:24.634589 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.634597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.635622 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.635627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.636882 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.636893 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.637757 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.637760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.637878 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.637918 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.638430 (+ 512us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.638511 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":27,"thread_start_us":140,"threads_started":1}
I20260810 22:59:24.638837 15336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.634346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33769 (local address 127.29.78.129:34221)
0810 22:59:24.634653 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.634666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:24.634800 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:24.635103 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:24.635107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:24.635125 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:24.635288 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.635295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.636313 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.636316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.637454 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:24.637464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.638398 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.638402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:24.638521 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.638544 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 22:59:24.638637 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 22:59:24.638702 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260810 22:59:24.638813 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.634404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:34221 (local address 127.29.78.131:33769)
0810 22:59:24.634646 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.634652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.634744 (+ 92us) server_negotiation.cc:407] Connection header received
0810 22:59:24.634900 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.634903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.634939 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.634981 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:24.635410 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.636043 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.637590 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.638166 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.638248 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.638300 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.638653 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.638716 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260810 22:59:24.639451 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf77561589184498932ac1b65319a5c2" candidate_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f34641c91474107b3b74dcbf7c4849f" is_pre_election: true
I20260810 22:59:24.639452 15112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf77561589184498932ac1b65319a5c2" candidate_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e19f60a15646099fc5574df30a96a6" is_pre_election: true
I20260810 22:59:24.639629 15112 raft_consensus.cc:2379] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a6d7a7d94814c9aba1b4753d7cd0f1c in current term 1: Already voted for candidate 5f34641c91474107b3b74dcbf7c4849f in this term.
I20260810 22:59:24.640257 15330 consensus_queue.cc:237] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:24.642841 14898 leader_election.cc:302] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c; no voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6
I20260810 22:59:24.643119 15332 raft_consensus.cc:2726] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 22:59:24.644160 14805 catalog_manager.cc:5495] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131). New cstate: current_term: 1 leader_uuid: "5f34641c91474107b3b74dcbf7c4849f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } health_report { overall_health: UNKNOWN } } }
I20260810 22:59:24.665735 15323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.662741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.29.78.131:33769)
0810 22:59:24.662851 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:24.662854 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:24.662883 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:59:24.662959 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:24.662961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:24.663011 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:24.663092 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:59:24.663295 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.663790 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.664560 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:24.664748 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:24.664816 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:24.665068 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:59:24.665447 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:59:24.665529 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:24.665586 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 22:59:24.665631 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 22:59:24.671120 15112 raft_consensus.cc:1261] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Refusing update from remote peer 5f34641c91474107b3b74dcbf7c4849f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:24.671161 14964 raft_consensus.cc:1261] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Refusing update from remote peer 5f34641c91474107b3b74dcbf7c4849f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:24.671952 15337 consensus_queue.cc:1035] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:24.672147 15330 consensus_queue.cc:1035] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:24.678781 15340 mvcc.cc:205] Tried to move back new op lower bound from 7317105724085985280 to 7317105723969220608. Current Snapshot: MvccSnapshot[applied={T|T < 7317105724085985280}]
I20260810 22:59:24.681073 15344 mvcc.cc:205] Tried to move back new op lower bound from 7317105724085985280 to 7317105723969220608. Current Snapshot: MvccSnapshot[applied={T|T < 7317105724085985280 or (T in {7317105724085985280})}]
I20260810 22:59:24.686801 14806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:24.686977 14806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:24.689287 14806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Sending DeleteTablet for 3 replicas of tablet bf77561589184498932ac1b65319a5c2
I20260810 22:59:24.690024 15237 tablet_service.cc:1514] Processing DeleteTablet for tablet bf77561589184498932ac1b65319a5c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:24 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:49212
I20260810 22:59:24.690037 15091 tablet_service.cc:1514] Processing DeleteTablet for tablet bf77561589184498932ac1b65319a5c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:24 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:60176
I20260810 22:59:24.690114 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 14875
I20260810 22:59:24.690078 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet bf77561589184498932ac1b65319a5c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:24 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:38758
I20260810 22:59:24.694532 15356 tablet_replica.cc:330] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: stopping tablet replica
I20260810 22:59:24.694852 15356 raft_consensus.cc:2229] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:24.695145 15356 raft_consensus.cc:2258] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:24.695487 15357 tablet_replica.cc:330] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: stopping tablet replica
I20260810 22:59:24.695724 15357 raft_consensus.cc:2229] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:59:24.696066 15357 raft_consensus.cc:2258] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:24.697113 15356 ts_tablet_manager.cc:1905] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:24.697741 15357 ts_tablet_manager.cc:1905] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:24.700492 15356 ts_tablet_manager.cc:1918] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:24.700592 15356 log.cc:1196] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/bf77561589184498932ac1b65319a5c2
I20260810 22:59:24.700520 15357 ts_tablet_manager.cc:1918] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:24.700775 15357 log.cc:1196] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/bf77561589184498932ac1b65319a5c2
I20260810 22:59:24.700950 15356 ts_tablet_manager.cc:1939] T bf77561589184498932ac1b65319a5c2 P b0e19f60a15646099fc5574df30a96a6: Deleting consensus metadata
I20260810 22:59:24.701201 15357 ts_tablet_manager.cc:1939] T bf77561589184498932ac1b65319a5c2 P 5f34641c91474107b3b74dcbf7c4849f: Deleting consensus metadata
W20260810 22:59:24.701273 14790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:38925 (error 108)
I20260810 22:59:24.702093 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 15021
I20260810 22:59:24.702176 14793 catalog_manager.cc:4841] TS b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739): tablet bf77561589184498932ac1b65319a5c2 (table test-table [id=f207e28fd593406088334beaf8138620]) successfully deleted
I20260810 22:59:24.702355 14793 catalog_manager.cc:4841] TS 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769): tablet bf77561589184498932ac1b65319a5c2 (table test-table [id=f207e28fd593406088334beaf8138620]) successfully deleted
I20260810 22:59:24.704545 15321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:24.704257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38925 (local address 127.0.0.1:49602)
0810 22:59:24.704370 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:24.704457 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 22:59:24.704784 14790 catalog_manager.cc:4584] TS 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129:38925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf77561589184498932ac1b65319a5c2: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38925: connect: Connection refused (error 111)
I20260810 22:59:24.709290 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 15168
I20260810 22:59:24.717250 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 14774
I20260810 22:59:44.726001 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35109
--webserver_interface=127.29.78.190
--webserver_port=38247
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 22:59:44.836802 15359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:44.837059 15359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:44.837106 15359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:44.840942 15359 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 22:59:44.840989 15359 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 22:59:44.841004 15359 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 22:59:44.841017 15359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 22:59:44.841051 15359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:44.841082 15359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:44.841096 15359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 22:59:44.841109 15359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 22:59:44.845993 15359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:44.846976 15359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:44.847883 15359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:44.861593 15366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:44.861557 15367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:44.861557 15370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:44.861629 15359 server_base.cc:1034] running on GCE node
I20260810 22:59:44.862591 15359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:44.863744 15359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:44.864941 15359 hybrid_clock.cc:648] HybridClock initialized: now 1786402784864930 us; error 31 us; skew 500 ppm
Aug 10 22:59:44 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402784, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:44.872000 15359 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 22:59:44.873011 15359 webserver.cc:460] Webserver started at http://127.29.78.190:38247/ using document root <none> and password file <none>
I20260810 22:59:44.873615 15359 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:44.873730 15359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:44.877637 15359 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:44.879873 15378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:44.881031 15359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:44.881174 15359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "635af0e94ecd4e279bd10ea03368dce3"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "41a25033c4e9d1591476a24dc3a2f172"
server_key_iv: "1b95dc53aa6942aaf9cd3cb59b912b78"
server_key_version: "encryptionkey@0"
I20260810 22:59:44.881780 15359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:44.906353 15359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:44.909153 15359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:44.909363 15359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:44.916661 15359 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35109
I20260810 22:59:44.916966 15433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35109 every 8 connection(s)
I20260810 22:59:44.917789 15359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 22:59:44.921789 15434 sys_catalog.cc:263] Verifying existing consensus state
I20260810 22:59:44.923244 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 15359
I20260810 22:59:44.923564 15434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Bootstrap starting.
I20260810 22:59:44.923758 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:38925
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:35109
--webserver_port=37295
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:59:44.940994 15434 log.cc:824] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:44.951777 15434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:59:44.952204 15434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Bootstrap complete.
I20260810 22:59:44.957628 15434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:44.958182 15434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635af0e94ecd4e279bd10ea03368dce3, State: Initialized, Role: FOLLOWER
I20260810 22:59:44.958746 15434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:44.958886 15434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 22:59:44.958986 15434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 22:59:44.959101 15434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 22:59:44.961575 15434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:44.961932 15434 leader_election.cc:302] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635af0e94ecd4e279bd10ea03368dce3; no voters:
I20260810 22:59:44.962303 15434 leader_election.cc:288] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 22:59:44.962455 15438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 22:59:44.962767 15438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 LEADER]: Becoming Leader. State: Replica: 635af0e94ecd4e279bd10ea03368dce3, State: Running, Role: LEADER
I20260810 22:59:44.963114 15438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 22:59:44.963594 15434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 22:59:44.964659 15440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635af0e94ecd4e279bd10ea03368dce3. Latest consensus state: current_term: 2 leader_uuid: "635af0e94ecd4e279bd10ea03368dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } } }
I20260810 22:59:44.964799 15440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:44.964659 15439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "635af0e94ecd4e279bd10ea03368dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } } }
I20260810 22:59:44.964999 15439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: This master's current role is: LEADER
I20260810 22:59:44.965761 15447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 22:59:44.970351 15447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 22:59:44.971035 15447 catalog_manager.cc:1259] Loaded cluster ID: 6736d0136f5e435f84ea787355a16659
I20260810 22:59:44.971093 15447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 22:59:44.973572 15447 catalog_manager.cc:1504] Loading token signing keys...
I20260810 22:59:44.974387 15447 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Loaded TSK: 0
I20260810 22:59:44.985683 15447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Generated new TSK 1
I20260810 22:59:44.985934 15447 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 22:59:45.044494 15436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:45.044735 15436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:45.044775 15436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:45.048669 15436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:45.048723 15436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:45.048779 15436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 22:59:45.053942 15436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:38925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=37295
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:45.054883 15436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:45.055857 15436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:45.065488 15464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:45.065615 15466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:45.065490 15463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:45.066023 15436 server_base.cc:1034] running on GCE node
I20260810 22:59:45.066392 15436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:45.067016 15436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:45.068203 15436 hybrid_clock.cc:648] HybridClock initialized: now 1786402785068184 us; error 34 us; skew 500 ppm
Aug 10 22:59:45 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402785, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:45.071008 15436 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 22:59:45.071977 15436 webserver.cc:460] Webserver started at http://127.29.78.129:37295/ using document root <none> and password file <none>
I20260810 22:59:45.072525 15436 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:45.072599 15436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:45.076406 15436 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:45.078604 15473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:45.079871 15436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:45.079993 15436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "d1a0ae08b6da9de2523a4d9325af3e3b"
server_key_iv: "af98b882e14008f1abcc10b9691ceedb"
server_key_version: "encryptionkey@0"
I20260810 22:59:45.080502 15436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:45.092792 15436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:45.095708 15436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:45.095901 15436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:45.096504 15436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:45.097621 15481 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 22:59:45.101380 15436 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 22:59:45.101438 15436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260810 22:59:45.101480 15436 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 22:59:45.103142 15436 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 22:59:45.103199 15436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 22:59:45.103492 15481 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap starting.
I20260810 22:59:45.113511 15436 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38925
I20260810 22:59:45.113521 15601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38925 every 8 connection(s)
I20260810 22:59:45.114653 15436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 22:59:45.120623 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 15436
I20260810 22:59:45.121137 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:44739
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:35109
--webserver_port=38903
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 22:59:45.123968 15481 log.cc:824] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Log is configured to *not* fsync() on all Append() calls
Aug 10 22:59:45 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402785, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:45.131919 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.116757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:53391)
0810 22:59:45.117264 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.117303 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.118169 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.119226 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.119233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.119635 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:45.120068 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.120081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.121470 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.121473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.122400 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.122410 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.123239 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.123243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.123442 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.124118 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:45.124136 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:45.125758 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:45.128230 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.128239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.128256 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.128675 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.129093 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.129097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.129099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.129300 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.129977 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:45.129983 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:45.130309 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.130523 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.130889 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":151,"threads_started":1}
I20260810 22:59:45.131963 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.116722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:53391 (local address 127.29.78.190:35109)
0810 22:59:45.117087 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.117120 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.117429 (+ 309us) server_negotiation.cc:407] Connection header received
0810 22:59:45.118564 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.118574 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.118863 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.119180 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:45.120238 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.121287 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.122530 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.123075 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.123255 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.125991 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:45.125997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:45.126012 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:45.126044 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:45.128080 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.128845 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.128850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.128855 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.128989 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.129428 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.129432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.129434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.129857 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:45.129981 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.130675 (+ 694us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.130942 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":119,"threads_started":1}
I20260810 22:59:45.132119 15481 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 22:59:45.132934 15481 tablet_bootstrap.cc:492] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap complete.
I20260810 22:59:45.134305 15481 ts_tablet_manager.cc:1397] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.007s
I20260810 22:59:45.134965 15602 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35109
I20260810 22:59:45.135183 15602 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:45.135659 15602 heartbeater.cc:511] Master 127.29.78.190:35109 requested a full tablet report, sending...
I20260810 22:59:45.136934 15481 raft_consensus.cc:348] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.137419 15481 raft_consensus.cc:729] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6d7a7d94814c9aba1b4753d7cd0f1c, State: Initialized, Role: FOLLOWER
I20260810 22:59:45.137934 15481 consensus_queue.cc:260] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.138285 15397 ts_manager.cc:194] Registered new tserver with Master: 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129:38925)
I20260810 22:59:45.138815 15481 ts_tablet_manager.cc:1428] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260810 22:59:45.142079 15397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:53391
I20260810 22:59:45.144347 15602 heartbeater.cc:503] Master 127.29.78.190:35109 was elected leader, sending a full tablet report...
Aug 10 22:59:45 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402784, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.129@KRBTEST.COM
I20260810 22:59:45.151474 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.141447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38925 (local address 127.0.0.1:52950)
0810 22:59:45.142151 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.142242 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.142382 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.142838 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.142843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.143268 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:45.143390 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.143398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.144674 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.144677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.145460 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.145469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.146219 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.146221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.146291 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.146804 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:45.146815 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:45.148012 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:45.149969 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.149976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.149980 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.150193 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.150507 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.150509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.150511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.150593 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.151038 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:45.151042 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:45.151126 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.151225 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.151351 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":78,"threads_started":1}
I20260810 22:59:45.151682 15610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.141502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.29.78.129:38925)
0810 22:59:45.141846 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.141886 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.142323 (+ 437us) server_negotiation.cc:407] Connection header received
0810 22:59:45.142536 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.142541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.142695 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.142837 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:45.143538 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.144553 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.145592 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.146082 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.146196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.148137 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:45.148147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:45.148156 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:45.148179 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:45.149850 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.150306 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.150312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.150314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.150412 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.150674 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.150677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.150679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.150935 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:45.151058 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.151325 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.151483 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":103,"threads_started":1}
I20260810 22:59:45.152644 15532 tablet_service.cc:1514] Processing DeleteTablet for tablet bf77561589184498932ac1b65319a5c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:24 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:52950
I20260810 22:59:45.153637 15611 tablet_replica.cc:330] stopping tablet replica
I20260810 22:59:45.153992 15611 raft_consensus.cc:2229] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:45.154254 15611 raft_consensus.cc:2258] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:45.155196 15611 ts_tablet_manager.cc:1905] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:45.158915 15611 ts_tablet_manager.cc:1918] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:45.159027 15611 log.cc:1196] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal/wals/bf77561589184498932ac1b65319a5c2
I20260810 22:59:45.159293 15611 ts_tablet_manager.cc:1939] T bf77561589184498932ac1b65319a5c2 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Deleting consensus metadata
I20260810 22:59:45.160832 15379 catalog_manager.cc:4841] TS 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129:38925): tablet bf77561589184498932ac1b65319a5c2 (table test-table [id=f207e28fd593406088334beaf8138620]) successfully deleted
W20260810 22:59:45.240059 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:45.240321 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:45.240363 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:45.244235 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:45.244288 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:45.244390 15606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 22:59:45.249159 15606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:44739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=38903
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:45.250237 15606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:45.251255 15606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:45.260462 15617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:45.260485 15616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:45.260540 15606 server_base.cc:1034] running on GCE node
W20260810 22:59:45.260511 15619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:45.261099 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:45.261792 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:45.262986 15606 hybrid_clock.cc:648] HybridClock initialized: now 1786402785262959 us; error 37 us; skew 500 ppm
Aug 10 22:59:45 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402785, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:45.266022 15606 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 22:59:45.266800 15606 webserver.cc:460] Webserver started at http://127.29.78.130:38903/ using document root <none> and password file <none>
I20260810 22:59:45.267310 15606 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:45.267416 15606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:45.270984 15606 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260810 22:59:45.273013 15629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:45.274243 15606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:45.274371 15606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "b0e19f60a15646099fc5574df30a96a6"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "b85d96632ff4a15b0f4486de505d8d01"
server_key_iv: "59e24190d4712cd659134bba1dfdcf80"
server_key_version: "encryptionkey@0"
I20260810 22:59:45.274890 15606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:45.291952 15606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:45.294785 15606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:45.295013 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:45.295568 15606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:45.296366 15606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:45.296437 15606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:45.296505 15606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:45.296548 15606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:45.305574 15606 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44739
I20260810 22:59:45.305594 15754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44739 every 8 connection(s)
I20260810 22:59:45.306566 15606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 22:59:45.308857 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 15606
I20260810 22:59:45.309350 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:33769
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:35109
--webserver_port=34827
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 22:59:45 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402785, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:45.321538 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.308404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:38831 (local address 127.29.78.190:35109)
0810 22:59:45.308549 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.308553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.309777 (+ 1224us) server_negotiation.cc:407] Connection header received
0810 22:59:45.310702 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.310705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.310764 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.310876 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:45.312102 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.312910 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.314221 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.314686 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.314817 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.317382 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:45.317387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:45.317390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:45.317427 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:45.319293 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.319793 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.319799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.319804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.319922 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.320265 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.320270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.320273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.320548 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:45.320665 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.321140 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.321302 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 22:59:45.322237 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.308838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:38831)
0810 22:59:45.309613 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.309674 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.310482 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.311058 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.311065 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.311421 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:45.311930 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.311942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.313049 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.313052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.314071 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.314094 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.314852 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.314854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.315003 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.315663 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:45.315680 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:45.317183 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:45.319414 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.319418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.319430 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.319687 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.320023 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.320025 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.320026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.320166 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.320656 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:45.320661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:45.320892 (+ 231us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.321047 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.321307 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":149,"threads_started":1}
I20260810 22:59:45.323521 15755 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35109
I20260810 22:59:45.323776 15755 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:45.324327 15755 heartbeater.cc:511] Master 127.29.78.190:35109 requested a full tablet report, sending...
I20260810 22:59:45.325402 15397 ts_manager.cc:194] Registered new tserver with Master: b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
I20260810 22:59:45.326041 15397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:38831
W20260810 22:59:45.425328 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 22:59:45.425590 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 22:59:45.425629 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 22:59:45.430023 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 22:59:45.430073 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 22:59:45.430122 15759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 22:59:45.435343 15759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:33769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=34827
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 22:59:45.436444 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 22:59:45.437455 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 22:59:45.447154 15765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:45.447154 15767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 22:59:45.447196 15764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 22:59:45.447260 15759 server_base.cc:1034] running on GCE node
I20260810 22:59:45.447952 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 22:59:45.448556 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 22:59:45.449733 15759 hybrid_clock.cc:648] HybridClock initialized: now 1786402785449723 us; error 33 us; skew 500 ppm
Aug 10 22:59:45 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402785, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 22:59:45.452401 15759 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 22:59:45.453265 15759 webserver.cc:460] Webserver started at http://127.29.78.131:34827/ using document root <none> and password file <none>
I20260810 22:59:45.453888 15759 fs_manager.cc:359] Metadata directory not provided
I20260810 22:59:45.454051 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 22:59:45.457790 15759 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 22:59:45.459865 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:45.460973 15759 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 22:59:45.461148 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "5f34641c91474107b3b74dcbf7c4849f"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "73be00c5a3218c448864fed9a2cde697"
server_key_iv: "759569c0a094e4271e7dbdd5bdbaa33e"
server_key_version: "encryptionkey@0"
I20260810 22:59:45.461673 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 22:59:45.480192 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 22:59:45.483042 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 22:59:45.483243 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 22:59:45.483952 15759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 22:59:45.484833 15759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 22:59:45.484958 15759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:45.485037 15759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 22:59:45.485087 15759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 22:59:45.494081 15759 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:33769
I20260810 22:59:45.494107 15899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:33769 every 8 connection(s)
I20260810 22:59:45.495064 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 22:59:45.497452 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 15759
Aug 10 22:59:45 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402785, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 22:59:45.508378 15903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.498150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54718 (local address 127.29.78.190:35109)
0810 22:59:45.498364 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.498368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.498642 (+ 274us) server_negotiation.cc:407] Connection header received
0810 22:59:45.498763 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.498765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.498808 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.498885 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:45.499814 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.500348 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.501389 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.501744 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.501895 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.503434 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:45.503438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:45.503440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:45.503468 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:45.505146 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.505621 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.505627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.505631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.505739 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.506515 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.506518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.506519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.506746 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:45.506976 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.508132 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.508254 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260810 22:59:45.511178 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.496985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:38413 (local address 127.29.78.190:35109)
0810 22:59:45.497145 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.497149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.498553 (+ 1404us) server_negotiation.cc:407] Connection header received
0810 22:59:45.499700 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.499702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.499733 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.499821 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:45.501014 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.501630 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.503416 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.503792 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.503873 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.506554 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:45.506557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:45.506559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:45.506585 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:45.508358 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.509154 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.509156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.509158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.509275 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.509916 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.509925 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.509926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.510102 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:45.510230 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.510892 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.511011 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 22:59:45.511989 15902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.497501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:38413)
0810 22:59:45.498390 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.498451 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.499496 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.499969 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.499975 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.500289 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0810 22:59:45.500863 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.500881 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.501784 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.501807 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.503253 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.503265 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.503945 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.503951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.504146 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.504789 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 22:59:45.504804 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 22:59:45.506375 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 22:59:45.508702 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.508708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.508719 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.509027 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.509566 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 22:59:45.509569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 22:59:45.509571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 22:59:45.509765 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 22:59:45.510397 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 22:59:45.510402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 22:59:45.510630 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.510827 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.511173 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":128,"threads_started":1}
I20260810 22:59:45.513072 15900 heartbeater.cc:347] Connected to a master server at 127.29.78.190:35109
I20260810 22:59:45.513319 15900 heartbeater.cc:464] Registering TS with master...
I20260810 22:59:45.513798 15900 heartbeater.cc:511] Master 127.29.78.190:35109 requested a full tablet report, sending...
I20260810 22:59:45.514858 15397 ts_manager.cc:194] Registered new tserver with Master: 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769)
I20260810 22:59:45.515399 15397 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:38413
I20260810 22:59:45.522244 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 22:59:45.526367 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.522857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.29.78.190:35109)
0810 22:59:45.523009 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.523014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.523200 (+ 186us) server_negotiation.cc:407] Connection header received
0810 22:59:45.523392 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.523395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.523486 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.523609 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:59:45.523843 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.524331 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.525195 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.525404 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.525509 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.525861 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:59:45.525988 (+ 127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 22:59:45.526243 (+ 255us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54722: token expired
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 22:59:45.526453 15604 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54722: token expired
W20260810 22:59:45.526733 15313 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.29.78.190:35109)
I20260810 22:59:45.534435 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.527088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.29.78.190:35109)
0810 22:59:45.527200 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.527204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.527240 (+ 36us) server_negotiation.cc:407] Connection header received
0810 22:59:45.527372 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.527375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.527421 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.527510 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 22:59:45.528137 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.528605 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.529405 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.529616 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.529725 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.531018 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 22:59:45.531022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 22:59:45.531024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 22:59:45.531049 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 22:59:45.532389 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.532929 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.532932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.532934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.533057 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 22:59:45.533397 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 22:59:45.533400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 22:59:45.533402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 22:59:45.533599 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 22:59:45.533742 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.534117 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.534259 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 22:59:45.536886 15397 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 22:59:45.540133 15397 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 22:59:45.547755 15532 tablet_service.cc:1467] Processing CreateTablet for tablet 17ca49c8b3674254acbd054c8f6df473 (DEFAULT_TABLE table=test-table [id=d9b3b40edeb7401b8957e5e7f31c3e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:45.548422 15532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ca49c8b3674254acbd054c8f6df473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:45.551734 15481 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap starting.
I20260810 22:59:45.552557 15910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.546934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44739 (local address 127.0.0.1:39346)
0810 22:59:45.547404 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.547435 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.547527 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.548226 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.548228 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.548246 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:45.548338 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.548343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.549496 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.549502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.550735 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.550746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.552224 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.552227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.552301 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.552336 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.552392 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.552446 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260810 22:59:45.552675 15481 tablet_bootstrap.cc:654] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:45.552723 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.547334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.29.78.130:44739)
0810 22:59:45.547782 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.547793 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.547818 (+ 25us) server_negotiation.cc:407] Connection header received
0810 22:59:45.547892 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.547899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.548101 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.548251 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:45.548433 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.549375 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.550877 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.552098 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.552225 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.552372 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.552429 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.552525 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":130,"threads_started":1}
I20260810 22:59:45.553742 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.546645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33769 (local address 127.0.0.1:50994)
0810 22:59:45.546773 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.546788 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.546863 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.547741 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.547744 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.547761 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:45.547861 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.547867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.549849 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.549852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.551012 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.551021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.553327 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.553331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.553431 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.553452 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.553534 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.553626 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":25}
I20260810 22:59:45.553920 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.546832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.29.78.131:33769)
0810 22:59:45.547201 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.547214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.547237 (+ 23us) server_negotiation.cc:407] Connection header received
0810 22:59:45.547313 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.547319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.547538 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.547742 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:45.548068 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.549697 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.551147 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.553197 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.553327 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.553463 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.553642 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.553739 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":142,"threads_started":1}
I20260810 22:59:45.554221 15481 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: No bootstrap required, opened a new log
I20260810 22:59:45.554318 15481 ts_tablet_manager.cc:1397] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 22:59:45.554771 15481 raft_consensus.cc:348] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.554890 15481 raft_consensus.cc:374] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:45.554931 15481 raft_consensus.cc:729] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6d7a7d94814c9aba1b4753d7cd0f1c, State: Initialized, Role: FOLLOWER
I20260810 22:59:45.555055 15481 consensus_queue.cc:260] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.555310 15481 ts_tablet_manager.cc:1428] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 22:59:45.555940 15684 tablet_service.cc:1467] Processing CreateTablet for tablet 17ca49c8b3674254acbd054c8f6df473 (DEFAULT_TABLE table=test-table [id=d9b3b40edeb7401b8957e5e7f31c3e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:45.556874 15684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ca49c8b3674254acbd054c8f6df473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:45.556912 15829 tablet_service.cc:1467] Processing CreateTablet for tablet 17ca49c8b3674254acbd054c8f6df473 (DEFAULT_TABLE table=test-table [id=d9b3b40edeb7401b8957e5e7f31c3e5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 22:59:45.558000 15829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ca49c8b3674254acbd054c8f6df473. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 22:59:45.563859 15913 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Bootstrap starting.
I20260810 22:59:45.564813 15914 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Bootstrap starting.
I20260810 22:59:45.566640 15913 tablet_bootstrap.cc:654] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:45.567233 15914 tablet_bootstrap.cc:654] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Neither blocks nor log segments found. Creating new log.
I20260810 22:59:45.567418 15913 log.cc:824] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:45.567945 15914 log.cc:824] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Log is configured to *not* fsync() on all Append() calls
I20260810 22:59:45.569500 15913 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: No bootstrap required, opened a new log
I20260810 22:59:45.569715 15913 ts_tablet_manager.cc:1397] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 22:59:45.569856 15914 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: No bootstrap required, opened a new log
I20260810 22:59:45.570108 15914 ts_tablet_manager.cc:1397] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 22:59:45.572716 15913 raft_consensus.cc:348] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.572944 15913 raft_consensus.cc:374] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:45.572980 15913 raft_consensus.cc:729] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e19f60a15646099fc5574df30a96a6, State: Initialized, Role: FOLLOWER
I20260810 22:59:45.572814 15914 raft_consensus.cc:348] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.573107 15914 raft_consensus.cc:374] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 22:59:45.573189 15914 raft_consensus.cc:729] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f34641c91474107b3b74dcbf7c4849f, State: Initialized, Role: FOLLOWER
I20260810 22:59:45.573473 15913 consensus_queue.cc:260] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.573609 15914 consensus_queue.cc:260] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.574159 15755 heartbeater.cc:503] Master 127.29.78.190:35109 was elected leader, sending a full tablet report...
I20260810 22:59:45.574252 15900 heartbeater.cc:503] Master 127.29.78.190:35109 was elected leader, sending a full tablet report...
I20260810 22:59:45.574501 15913 ts_tablet_manager.cc:1428] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 22:59:45.575500 15914 ts_tablet_manager.cc:1428] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260810 22:59:45.615906 15603 tablet.cc:2369] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:45.680166 15608 raft_consensus.cc:482] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 22:59:45.680404 15608 raft_consensus.cc:504] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.681569 15608 leader_election.cc:288] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769), b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
I20260810 22:59:45.686466 15911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.682572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54497 (local address 127.29.78.130:44739)
0810 22:59:45.682726 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.682732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.682774 (+ 42us) server_negotiation.cc:407] Connection header received
0810 22:59:45.682924 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.682927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.682981 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.683074 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:45.683428 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.683925 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.685148 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.685764 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.685934 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.685995 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.686241 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.686336 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 22:59:45.686491 15920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.682380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44739 (local address 127.29.78.129:54497)
0810 22:59:45.682695 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.682706 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.682814 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.683078 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.683082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.683095 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:45.683307 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.683313 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.684044 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.684049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.685017 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.685027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.685945 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.685948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.686035 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.686095 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.686213 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.686296 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":23,"thread_start_us":66,"threads_started":1}
I20260810 22:59:45.686623 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.682061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:54001 (local address 127.29.78.131:33769)
0810 22:59:45.682209 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.682213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.682409 (+ 196us) server_negotiation.cc:407] Connection header received
0810 22:59:45.682637 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.682641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.682698 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.682755 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 22:59:45.683326 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.683948 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.685311 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.685965 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.686089 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.686155 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.686411 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.686490 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 22:59:45.686955 15919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.681980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33769 (local address 127.29.78.129:54001)
0810 22:59:45.682300 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.682318 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 22:59:45.682445 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 22:59:45.682961 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 22:59:45.682964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 22:59:45.682984 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 22:59:45.683206 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.683214 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.684062 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.684067 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.685152 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 22:59:45.685166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.686078 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.686081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 22:59:45.686236 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.686256 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 22:59:45.686389 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 22:59:45.686463 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":65,"thread_start_us":85,"threads_started":1}
I20260810 22:59:45.687297 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ca49c8b3674254acbd054c8f6df473" candidate_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e19f60a15646099fc5574df30a96a6" is_pre_election: true
I20260810 22:59:45.687541 15851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ca49c8b3674254acbd054c8f6df473" candidate_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f34641c91474107b3b74dcbf7c4849f" is_pre_election: true
I20260810 22:59:45.687629 15706 raft_consensus.cc:2454] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6d7a7d94814c9aba1b4753d7cd0f1c in term 0.
I20260810 22:59:45.687832 15851 raft_consensus.cc:2454] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a6d7a7d94814c9aba1b4753d7cd0f1c in term 0.
I20260810 22:59:45.688192 15477 leader_election.cc:302] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c, b0e19f60a15646099fc5574df30a96a6; no voters:
I20260810 22:59:45.688455 15608 raft_consensus.cc:2781] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 22:59:45.688544 15608 raft_consensus.cc:482] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 22:59:45.688594 15608 raft_consensus.cc:3037] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:45.689651 15608 raft_consensus.cc:504] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.690062 15608 leader_election.cc:288] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 1 election: Requested vote from peers 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769), b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
I20260810 22:59:45.690395 15851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ca49c8b3674254acbd054c8f6df473" candidate_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f34641c91474107b3b74dcbf7c4849f"
I20260810 22:59:45.690544 15706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ca49c8b3674254acbd054c8f6df473" candidate_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e19f60a15646099fc5574df30a96a6"
I20260810 22:59:45.690543 15851 raft_consensus.cc:3037] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:45.690667 15706 raft_consensus.cc:3037] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 22:59:45.691973 15706 raft_consensus.cc:2454] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6d7a7d94814c9aba1b4753d7cd0f1c in term 1.
I20260810 22:59:45.691964 15851 raft_consensus.cc:2454] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a6d7a7d94814c9aba1b4753d7cd0f1c in term 1.
I20260810 22:59:45.692293 15477 leader_election.cc:302] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c, 5f34641c91474107b3b74dcbf7c4849f; no voters:
I20260810 22:59:45.692489 15608 raft_consensus.cc:2781] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 22:59:45.692718 15608 raft_consensus.cc:686] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 LEADER]: Becoming Leader. State: Replica: 4a6d7a7d94814c9aba1b4753d7cd0f1c, State: Running, Role: LEADER
I20260810 22:59:45.693007 15608 consensus_queue.cc:237] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 22:59:45.695982 15397 catalog_manager.cc:5495] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129). New cstate: current_term: 1 leader_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } health_report { overall_health: HEALTHY } } }
I20260810 22:59:45.742264 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.738796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.29.78.129:38925)
0810 22:59:45.739180 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 22:59:45.739186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 22:59:45.739215 (+ 29us) server_negotiation.cc:407] Connection header received
0810 22:59:45.739291 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 22:59:45.739293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 22:59:45.739349 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 22:59:45.739453 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 22:59:45.739693 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.740209 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.740969 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 22:59:45.741193 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 22:59:45.741261 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 22:59:45.741475 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 22:59:45.741890 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 22:59:45.741969 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 22:59:45.742079 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 22:59:45.742126 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":197,"threads_started":1}
W20260810 22:59:45.746031 15901 tablet.cc:2369] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 22:59:45.746240 15706 raft_consensus.cc:1261] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Refusing update from remote peer 4a6d7a7d94814c9aba1b4753d7cd0f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:45.746237 15851 raft_consensus.cc:1261] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 1 FOLLOWER]: Refusing update from remote peer 4a6d7a7d94814c9aba1b4753d7cd0f1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 22:59:45.747009 15608 consensus_queue.cc:1035] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:45.747172 15921 consensus_queue.cc:1035] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 22:59:45.755366 15927 mvcc.cc:205] Tried to move back new op lower bound from 7317105810409951232 to 7317105810201202688. Current Snapshot: MvccSnapshot[applied={T|T < 7317105810409951232}]
I20260810 22:59:45.761745 15397 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:45.761981 15397 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 22:59:45.764343 15397 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Sending DeleteTablet for 3 replicas of tablet 17ca49c8b3674254acbd054c8f6df473
I20260810 22:59:45.765125 15532 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ca49c8b3674254acbd054c8f6df473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:45 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:52950
I20260810 22:59:45.765189 15684 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ca49c8b3674254acbd054c8f6df473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:45 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:39346
I20260810 22:59:45.765293 15829 tablet_service.cc:1514] Processing DeleteTablet for tablet 17ca49c8b3674254acbd054c8f6df473 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 22:59:45 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:50994
I20260810 22:59:45.765738 15938 tablet_replica.cc:330] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: stopping tablet replica
I20260810 22:59:45.766081 15938 raft_consensus.cc:2229] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:45.766117 15937 tablet_replica.cc:330] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: stopping tablet replica
I20260810 22:59:45.766110 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 15436
I20260810 22:59:45.766244 15937 raft_consensus.cc:2229] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 LEADER]: Raft consensus shutting down.
I20260810 22:59:45.766397 15938 raft_consensus.cc:2258] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:45.766784 15939 tablet_replica.cc:330] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: stopping tablet replica
I20260810 22:59:45.767009 15939 raft_consensus.cc:2229] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 22:59:45.767220 15939 raft_consensus.cc:2258] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 22:59:45.769662 15938 ts_tablet_manager.cc:1905] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:45.772439 15939 ts_tablet_manager.cc:1905] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 22:59:45.773351 15938 ts_tablet_manager.cc:1918] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:45.773475 15938 log.cc:1196] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/17ca49c8b3674254acbd054c8f6df473
I20260810 22:59:45.773864 15938 ts_tablet_manager.cc:1939] T 17ca49c8b3674254acbd054c8f6df473 P b0e19f60a15646099fc5574df30a96a6: Deleting consensus metadata
I20260810 22:59:45.774850 15939 ts_tablet_manager.cc:1918] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 22:59:45.774957 15939 log.cc:1196] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/17ca49c8b3674254acbd054c8f6df473
I20260810 22:59:45.775245 15382 catalog_manager.cc:4841] TS b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739): tablet 17ca49c8b3674254acbd054c8f6df473 (table test-table [id=d9b3b40edeb7401b8957e5e7f31c3e5d]) successfully deleted
I20260810 22:59:45.775310 15939 ts_tablet_manager.cc:1939] T 17ca49c8b3674254acbd054c8f6df473 P 5f34641c91474107b3b74dcbf7c4849f: Deleting consensus metadata
I20260810 22:59:45.776276 15382 catalog_manager.cc:4841] TS 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769): tablet 17ca49c8b3674254acbd054c8f6df473 (table test-table [id=d9b3b40edeb7401b8957e5e7f31c3e5d]) successfully deleted
W20260810 22:59:45.778599 15379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:38925 (error 108)
I20260810 22:59:45.779016 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 15606
I20260810 22:59:45.782750 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 22:59:45.779758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38925 (local address 127.0.0.1:52960)
0810 22:59:45.782605 (+ 2847us) negotiation.cc:107] Waiting for socket to connect
0810 22:59:45.782673 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2759}
W20260810 22:59:45.782918 15379 catalog_manager.cc:4584] TS 4a6d7a7d94814c9aba1b4753d7cd0f1c (127.29.78.129:38925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17ca49c8b3674254acbd054c8f6df473: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38925: connect: Connection refused (error 111)
I20260810 22:59:45.787245 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 15759
I20260810 22:59:45.795313 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 15359
I20260810 23:00:05.807181 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35109
--webserver_interface=127.29.78.190
--webserver_port=38247
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:35109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:05.919034 15941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:05.919299 15941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:05.919340 15941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:05.923326 15941 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:00:05.923373 15941 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:00:05.923388 15941 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:00:05.923401 15941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:05.923418 15941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:05.923489 15941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:05.923511 15941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:05.923539 15941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:05.928714 15941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:35109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=38247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:05.929741 15941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:05.930781 15941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:05.939251 15947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:05.939301 15950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:05.939491 15941 server_base.cc:1034] running on GCE node
W20260810 23:00:05.939518 15948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:05.940147 15941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:05.941205 15941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:05.942365 15941 hybrid_clock.cc:648] HybridClock initialized: now 1786402805942326 us; error 49 us; skew 500 ppm
Aug 10 23:00:05 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402805, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:05.950840 15941 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:05.951931 15941 webserver.cc:460] Webserver started at http://127.29.78.190:38247/ using document root <none> and password file <none>
I20260810 23:00:05.952543 15941 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:05.952646 15941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:05.956631 15941 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:05.958945 15961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:05.960178 15941 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:05.960323 15941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "635af0e94ecd4e279bd10ea03368dce3"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "41a25033c4e9d1591476a24dc3a2f172"
server_key_iv: "1b95dc53aa6942aaf9cd3cb59b912b78"
server_key_version: "encryptionkey@0"
I20260810 23:00:05.960870 15941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:05.971033 15941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:05.974143 15941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:05.974364 15941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:05.981595 15941 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:35109
I20260810 23:00:05.981999 16015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:35109 every 8 connection(s)
I20260810 23:00:05.982774 15941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:05.984352 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 15941
I20260810 23:00:05.984860 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:38925
--local_ip_for_outbound_sockets=127.29.78.129
--tserver_master_addrs=127.29.78.190:35109
--webserver_port=37295
--webserver_interface=127.29.78.129
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:00:05.987738 16016 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:00:05.989357 16016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Bootstrap starting.
I20260810 23:00:06.007453 16016 log.cc:824] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:06.018210 16016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:00:06.018606 16016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Bootstrap complete.
I20260810 23:00:06.022488 16016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 23:00:06.022821 16016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635af0e94ecd4e279bd10ea03368dce3, State: Initialized, Role: FOLLOWER
I20260810 23:00:06.023272 16016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 23:00:06.023391 16016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:06.023509 16016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:06.023617 16016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:00:06.025532 16016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 23:00:06.025959 16016 leader_election.cc:302] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635af0e94ecd4e279bd10ea03368dce3; no voters:
I20260810 23:00:06.026332 16016 leader_election.cc:288] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:00:06.026448 16021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:00:06.026715 16021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [term 3 LEADER]: Becoming Leader. State: Replica: 635af0e94ecd4e279bd10ea03368dce3, State: Running, Role: LEADER
I20260810 23:00:06.027063 16021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } }
I20260810 23:00:06.027616 16016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:06.028041 16021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635af0e94ecd4e279bd10ea03368dce3. Latest consensus state: current_term: 3 leader_uuid: "635af0e94ecd4e279bd10ea03368dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } } }
I20260810 23:00:06.028179 16021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:06.028514 16022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "635af0e94ecd4e279bd10ea03368dce3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635af0e94ecd4e279bd10ea03368dce3" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 35109 } } }
I20260810 23:00:06.028692 16022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:06.028611 16025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:06.033757 16025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:06.034487 16025 catalog_manager.cc:1259] Loaded cluster ID: 6736d0136f5e435f84ea787355a16659
I20260810 23:00:06.034551 16025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:06.036911 16025 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:06.037655 16025 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Loaded TSK: 0
I20260810 23:00:06.037747 16025 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 635af0e94ecd4e279bd10ea03368dce3: Loaded TSK: 1
F20260810 23:00:06.038475 16025 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f343ac7c9ad google::LogMessage::Fail() at ??:0
@ 0x7f343ac7f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f343ac7c4de google::LogMessage::Flush() at ??:0
@ 0x7f343ac7da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f343d94018d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f343d938f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f34455e17bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f34455bb99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f34455bc178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f34455fddf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f344561c82b std::function<>::operator()() at ??:0
@ 0x7f34455bbb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f34455bcf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f34455b9579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f34455fd640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f343cece3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f343cecec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f343ced03f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f343cebf512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f343e5036db start_thread at ??:0
@ 0x7f343977a61f clone at ??:0
W20260810 23:00:06.134773 16018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:06.135005 16018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:06.135051 16018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:06.138935 16018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:06.138983 16018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:06.139066 16018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 23:00:06.144047 16018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:38925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=37295
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:06.145061 16018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:06.146148 16018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:06.155802 16068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:06.155867 16069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:06.156105 16018 server_base.cc:1034] running on GCE node
W20260810 23:00:06.155833 16072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:06.156606 16018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:06.157248 16018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:06.158437 16018 hybrid_clock.cc:648] HybridClock initialized: now 1786402806158414 us; error 41 us; skew 500 ppm
Aug 10 23:00:06 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:06.161404 16018 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 23:00:06.162434 16018 webserver.cc:460] Webserver started at http://127.29.78.129:37295/ using document root <none> and password file <none>
I20260810 23:00:06.163012 16018 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:06.163077 16018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:06.166924 16018 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 23:00:06.169198 16080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:06.170506 16018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:06.170635 16018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c"
format_stamp: "Formatted at 2026-08-10 22:59:23 on dist-test-slave-6nmv"
server_key: "d1a0ae08b6da9de2523a4d9325af3e3b"
server_key_iv: "af98b882e14008f1abcc10b9691ceedb"
server_key_version: "encryptionkey@0"
I20260810 23:00:06.171147 16018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:06.196964 16018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:06.200327 16018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:06.200538 16018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:06.201159 16018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:06.202437 16088 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:00:06.207818 16018 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:00:06.207926 16018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 23:00:06.208009 16018 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:00:06.209654 16018 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:00:06.209729 16018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 23:00:06.210074 16088 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap starting.
I20260810 23:00:06.220700 16018 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38925
I20260810 23:00:06.220917 16203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38925 every 8 connection(s)
I20260810 23:00:06.221719 16018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 23:00:06.222595 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 16018
I20260810 23:00:06.223026 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:44739
--local_ip_for_outbound_sockets=127.29.78.130
--tserver_master_addrs=127.29.78.190:35109
--webserver_port=38903
--webserver_interface=127.29.78.130
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:00:06.232168 16088 log.cc:824] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:06.241240 16088 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:00:06.241667 16088 tablet_bootstrap.cc:492] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Bootstrap complete.
I20260810 23:00:06.243207 16088 ts_tablet_manager.cc:1397] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20260810 23:00:06.246363 16088 raft_consensus.cc:348] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 23:00:06.246923 16088 raft_consensus.cc:729] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a6d7a7d94814c9aba1b4753d7cd0f1c, State: Initialized, Role: FOLLOWER
I20260810 23:00:06.247522 16088 consensus_queue.cc:260] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 23:00:06.248606 16088 ts_tablet_manager.cc:1428] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260810 23:00:06.269398 16207 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:06.224822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:54983)
0810 23:00:06.226440 (+ 1618us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.226511 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:06.227267 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:06.268629 (+ 41362us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1366,"thread_start_us":142,"threads_started":1}
W20260810 23:00:06.270027 16204 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:35109 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:35109: Client connection negotiation failed: client connection to 127.29.78.190:35109: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:00:06.270812 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.270564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:58101)
0810 23:00:06.270698 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.270760 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:00:06.271512 16207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.271288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:55133)
0810 23:00:06.271401 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.271467 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:00:06.271727 16204 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:00:06.340669 16206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:06.340978 16206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:06.341059 16206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:06.345027 16206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:06.345132 16206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:06.345228 16206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 23:00:06.350230 16206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:44739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=38903
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:06.351331 16206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:06.352577 16206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:06.362140 16216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:06.362206 16218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:06.362210 16215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:06.362424 16206 server_base.cc:1034] running on GCE node
I20260810 23:00:06.362826 16206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:06.363441 16206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:06.364629 16206 hybrid_clock.cc:648] HybridClock initialized: now 1786402806364596 us; error 41 us; skew 500 ppm
Aug 10 23:00:06 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:06.367595 16206 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 23:00:06.368535 16206 webserver.cc:460] Webserver started at http://127.29.78.130:38903/ using document root <none> and password file <none>
I20260810 23:00:06.369135 16206 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:06.369243 16206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:06.373220 16206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:00:06.375542 16225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:06.376793 16206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:06.376973 16206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "b0e19f60a15646099fc5574df30a96a6"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "b85d96632ff4a15b0f4486de505d8d01"
server_key_iv: "59e24190d4712cd659134bba1dfdcf80"
server_key_version: "encryptionkey@0"
I20260810 23:00:06.377517 16206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:06.388032 16206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:06.391345 16206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:06.391549 16206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:06.392148 16206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:06.392979 16206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:06.393052 16206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:06.393121 16206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:06.393154 16206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:06.402359 16206 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44739
I20260810 23:00:06.402381 16348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44739 every 8 connection(s)
I20260810 23:00:06.403326 16206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 23:00:06.407006 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.405511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:55997)
0810 23:00:06.406221 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.406326 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":107,"threads_started":1}
W20260810 23:00:06.407516 16349 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:35109 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:35109: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
I20260810 23:00:06.408165 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.407903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:40585)
0810 23:00:06.408056 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.408114 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:00:06.408913 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.408684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:53705)
0810 23:00:06.408802 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.408848 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 23:00:06.409094 16349 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:00:06.412241 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 16206
I20260810 23:00:06.412684 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:33769
--local_ip_for_outbound_sockets=127.29.78.131
--tserver_master_addrs=127.29.78.190:35109
--webserver_port=34827
--webserver_interface=127.29.78.131
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:06.525929 16353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:06.526197 16353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:06.526242 16353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:06.530218 16353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:06.530274 16353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:06.530395 16353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 23:00:06.535496 16353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:33769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=34827
--tserver_master_addrs=127.29.78.190:35109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:06.536552 16353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:06.537564 16353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:06.547171 16358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:06.547209 16359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:06.547210 16361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:06.547534 16353 server_base.cc:1034] running on GCE node
I20260810 23:00:06.547948 16353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:06.548547 16353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:06.549721 16353 hybrid_clock.cc:648] HybridClock initialized: now 1786402806549696 us; error 33 us; skew 500 ppm
Aug 10 23:00:06 dist-test-slave-6nmv krb5kdc[14760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:06.552800 16353 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 23:00:06.553689 16353 webserver.cc:460] Webserver started at http://127.29.78.131:34827/ using document root <none> and password file <none>
I20260810 23:00:06.554301 16353 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:06.554418 16353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:06.558204 16353 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 23:00:06.560372 16368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:06.561604 16353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:06.561777 16353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "5f34641c91474107b3b74dcbf7c4849f"
format_stamp: "Formatted at 2026-08-10 22:59:24 on dist-test-slave-6nmv"
server_key: "73be00c5a3218c448864fed9a2cde697"
server_key_iv: "759569c0a094e4271e7dbdd5bdbaa33e"
server_key_version: "encryptionkey@0"
I20260810 23:00:06.562397 16353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:06.572525 16353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:06.576188 16353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:06.576416 16353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:06.577051 16353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:06.577991 16353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:06.578076 16353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:06.578150 16353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:06.578184 16353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:06.588151 16353 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:33769
I20260810 23:00:06.588181 16493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:33769 every 8 connection(s)
I20260810 23:00:06.589175 16353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 23:00:06.589883 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 16353
I20260810 23:00:06.593078 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.591606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:39611)
0810 23:00:06.592276 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.592375 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":116,"threads_started":1}
W20260810 23:00:06.593503 16494 heartbeater.cc:644] Failed to heartbeat to 127.29.78.190:35109 (0 consecutive failures): Network error: Failed to ping master at 127.29.78.190:35109: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
I20260810 23:00:06.594164 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.593949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:57841)
0810 23:00:06.594056 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.594105 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 23:00:06.594792 16498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:06.594582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:43047)
0810 23:00:06.594693 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:06.594737 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:00:06.594961 16494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:00:07.272784 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:07.272320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:55213)
0810 23:00:07.272608 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:07.272668 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":98,"threads_started":1}
I20260810 23:00:07.410125 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:07.409657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:43707)
0810 23:00:07.409972 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:07.410027 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":67,"threads_started":1}
I20260810 23:00:07.595955 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:07.595453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:37245)
0810 23:00:07.595769 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:07.595830 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":80,"threads_started":1}
I20260810 23:00:07.877630 16502 raft_consensus.cc:482] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:07.877928 16502 raft_consensus.cc:504] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 23:00:07.879077 16502 leader_election.cc:288] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769), b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
Aug 10 23:00:07 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.130@KRBTEST.COM
Aug 10 23:00:07 dist-test-slave-6nmv krb5kdc[14760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402806, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.131@KRBTEST.COM
I20260810 23:00:07.895347 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:07.879994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:38049 (local address 127.29.78.130:44739)
0810 23:00:07.880297 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:07.880327 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:07.880413 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:00:07.881137 (+ 724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:07.881144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:07.881553 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:07.881867 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:07.883048 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.884099 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.886019 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.886733 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.886912 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:07.890135 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:07.890142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:07.890166 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:07.890194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:07.892608 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:07.893181 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:07.893186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:07.893190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:07.893307 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:07.893679 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:07.893687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:07.893688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:07.894142 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:07.894317 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:07.894852 (+ 535us) server_negotiation.cc:299] Negotiation successful
0810 23:00:07.895147 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":130,"threads_started":1}
I20260810 23:00:07.895334 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:07.879759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44739 (local address 127.29.78.129:38049)
0810 23:00:07.880080 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:07.880093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:07.880220 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:07.881780 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:07.881784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:07.882331 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:07.882851 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:07.882866 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.884262 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.884269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:07.885873 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:07.885884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.886871 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.886874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:07.887140 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:07.887923 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:07.887971 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:07.889873 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:07.892739 (+ 2866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:07.892744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:07.892754 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:07.893034 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:07.893414 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:07.893418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:07.893420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:07.893562 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:07.894296 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:07.894302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:07.894528 (+ 226us) client_negotiation.cc:769] Sending connection context
0810 23:00:07.894714 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 23:00:07.895160 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":493,"spinlock_wait_cycles":61440,"thread_start_us":71,"threads_started":1}
I20260810 23:00:07.895813 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:07.879358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33769 (local address 127.29.78.129:42941)
0810 23:00:07.879640 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:07.879657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:07.879758 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:07.881651 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:07.881657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:07.882336 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:07.882851 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:07.882865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.884365 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.884387 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:07.885873 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:07.885884 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.886871 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.886874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:07.887142 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:07.887923 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:07.887953 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:07.890285 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:07.893077 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:07.893081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:07.893083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:07.893285 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:07.893655 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:07.893658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:07.893660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:07.893758 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:07.894419 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:07.894422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:07.894537 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 23:00:07.894714 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 23:00:07.895068 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":230,"spinlock_wait_cycles":19968,"thread_start_us":113,"threads_started":1}
I20260810 23:00:07.895955 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:07.879542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:42941 (local address 127.29.78.131:33769)
0810 23:00:07.879966 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:07.879996 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:07.880045 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:00:07.880917 (+ 872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:07.880924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:07.881273 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:07.881603 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:07.883047 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.884206 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.886011 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:07.886733 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:07.886940 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:07.890484 (+ 3544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:07.890493 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:07.890510 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:07.890843 (+ 333us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:07.892956 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:07.893410 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:07.893415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:07.893420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:07.893548 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:07.893884 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:07.893887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:07.893896 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:07.894278 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:07.894452 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:07.895470 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0810 23:00:07.895758 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":60,"threads_started":1}
W20260810 23:00:07.898182 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
W20260810 23:00:07.898782 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
I20260810 23:00:07.898933 16084 leader_election.cc:302] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c; no voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6
I20260810 23:00:07.899221 16502 raft_consensus.cc:2726] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:00:08.273803 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:08.273543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:39025)
0810 23:00:08.273668 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:08.273725 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:00:08.411298 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:08.410893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:48077)
0810 23:00:08.411138 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:08.411201 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":70,"threads_started":1}
I20260810 23:00:08.597226 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:08.596775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:55813)
0810 23:00:08.597051 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:08.597131 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":78,"threads_started":1}
I20260810 23:00:09.275158 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:09.274709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:51367)
0810 23:00:09.274982 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:09.275050 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260810 23:00:09.412532 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:09.412102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:57611)
0810 23:00:09.412353 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:09.412416 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260810 23:00:09.598567 16516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:09.598103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:59213)
0810 23:00:09.598358 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:09.598416 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":74,"threads_started":1}
I20260810 23:00:09.746687 16517 raft_consensus.cc:482] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:09.746836 16517 raft_consensus.cc:504] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 23:00:09.747342 16517 leader_election.cc:288] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769), b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
W20260810 23:00:09.748286 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
W20260810 23:00:09.748415 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
I20260810 23:00:09.748456 16084 leader_election.cc:302] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c; no voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6
I20260810 23:00:09.748597 16517 raft_consensus.cc:2726] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:00:10.276288 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:10.275893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:59513)
0810 23:00:10.276141 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:10.276194 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260810 23:00:10.413726 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:10.413308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:41555)
0810 23:00:10.413577 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:10.413639 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":103,"threads_started":1}
I20260810 23:00:10.599778 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:10.599324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:37633)
0810 23:00:10.599635 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:10.599688 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
I20260810 23:00:11.277872 16522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:11.277264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:54893)
0810 23:00:11.277643 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:11.277711 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":93,"threads_started":1}
I20260810 23:00:11.415055 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:11.414605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:44833)
0810 23:00:11.414895 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:11.414957 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
I20260810 23:00:11.601006 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:11.600573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:38683)
0810 23:00:11.600835 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:11.600889 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":95,"threads_started":1}
I20260810 23:00:12.279027 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:12.278582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:40563)
0810 23:00:12.278863 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:12.278924 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":97,"threads_started":1}
I20260810 23:00:12.416191 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:12.415777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:58917)
0810 23:00:12.416062 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:12.416111 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
I20260810 23:00:12.602109 16527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:12.601626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:42515)
0810 23:00:12.601918 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:12.601993 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":82,"threads_started":1}
I20260810 23:00:12.849255 16528 raft_consensus.cc:482] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:12.849421 16528 raft_consensus.cc:504] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 23:00:12.849915 16528 leader_election.cc:288] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769), b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
W20260810 23:00:12.850739 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
W20260810 23:00:12.850869 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
I20260810 23:00:12.851022 16084 leader_election.cc:302] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c; no voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6
I20260810 23:00:12.851172 16528 raft_consensus.cc:2726] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:00:13.280274 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:13.279785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:33793)
0810 23:00:13.280097 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:13.280158 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":124,"threads_started":1}
I20260810 23:00:13.417344 16530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:13.416904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:37583)
0810 23:00:13.417199 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:13.417255 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":70,"threads_started":1}
I20260810 23:00:13.603384 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:13.602928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:57951)
0810 23:00:13.603218 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:13.603276 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260810 23:00:14.281591 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:14.281157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:49019)
0810 23:00:14.281404 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:14.281463 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260810 23:00:14.418606 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:14.418141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:56111)
0810 23:00:14.418450 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:14.418505 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":82,"threads_started":1}
I20260810 23:00:14.604638 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:14.604146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:39969)
0810 23:00:14.604449 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:14.604516 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":92,"threads_started":1}
I20260810 23:00:14.888968 16535 raft_consensus.cc:482] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:14.889106 16535 raft_consensus.cc:504] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 23:00:14.889564 16535 leader_election.cc:288] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769), b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
W20260810 23:00:14.890555 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
W20260810 23:00:14.890698 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
I20260810 23:00:14.890760 16084 leader_election.cc:302] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c; no voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6
I20260810 23:00:14.890980 16535 raft_consensus.cc:2726] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:00:15.282716 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:15.282294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:45561)
0810 23:00:15.282565 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:15.282619 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260810 23:00:15.419762 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:15.419352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:44555)
0810 23:00:15.419598 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:15.419656 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260810 23:00:15.605768 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:15.605380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:50675)
0810 23:00:15.605629 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:15.605680 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
I20260810 23:00:16.283887 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:16.283502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:41407)
0810 23:00:16.283740 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:16.283792 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":83,"threads_started":1}
I20260810 23:00:16.421121 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:16.420643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:35619)
0810 23:00:16.420933 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:16.420996 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260810 23:00:16.607029 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:16.606559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:37943)
0810 23:00:16.606825 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:16.606891 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":73,"threads_started":1}
I20260810 23:00:16.747253 16542 raft_consensus.cc:482] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:16.747444 16542 raft_consensus.cc:504] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f34641c91474107b3b74dcbf7c4849f" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33769 } } peers { permanent_uuid: "b0e19f60a15646099fc5574df30a96a6" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44739 } } peers { permanent_uuid: "4a6d7a7d94814c9aba1b4753d7cd0f1c" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38925 } }
I20260810 23:00:16.747972 16542 leader_election.cc:288] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769), b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739)
W20260810 23:00:16.748845 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5f34641c91474107b3b74dcbf7c4849f (127.29.78.131:33769): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
W20260810 23:00:16.749128 16084 leader_election.cc:341] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0e19f60a15646099fc5574df30a96a6 (127.29.78.130:44739): Not found: Tablet not found: 17ca49c8b3674254acbd054c8f6df473
I20260810 23:00:16.749225 16084 leader_election.cc:302] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a6d7a7d94814c9aba1b4753d7cd0f1c; no voters: 5f34641c91474107b3b74dcbf7c4849f, b0e19f60a15646099fc5574df30a96a6
I20260810 23:00:16.749529 16542 raft_consensus.cc:2726] T 17ca49c8b3674254acbd054c8f6df473 P 4a6d7a7d94814c9aba1b4753d7cd0f1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:00:17.284952 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:17.284584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:45441)
0810 23:00:17.284813 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:17.284862 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260810 23:00:17.422596 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:17.422106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:32943)
0810 23:00:17.422429 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:17.422498 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":87,"threads_started":1}
I20260810 23:00:17.608090 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:17.607718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:59919)
0810 23:00:17.607950 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:17.607998 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260810 23:00:18.286224 16547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:18.285755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:48859)
0810 23:00:18.286068 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:18.286127 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":127,"threads_started":1}
I20260810 23:00:18.423988 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:18.423570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:55955)
0810 23:00:18.423842 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:18.423896 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
I20260810 23:00:18.609324 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:18.608793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:36283)
0810 23:00:18.609110 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:18.609174 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":73,"threads_started":1}
I20260810 23:00:19.287313 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:19.286932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:43247)
0810 23:00:19.287173 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:19.287221 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
I20260810 23:00:19.425158 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:19.424733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:57039)
0810 23:00:19.424992 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:19.425052 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":72,"threads_started":1}
I20260810 23:00:19.610589 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:19.610134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:36093)
0810 23:00:19.610416 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:19.610491 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":96,"threads_started":1}
I20260810 23:00:20.288463 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:20.288062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:46035)
0810 23:00:20.288311 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:20.288362 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260810 23:00:20.426367 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:20.425917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:33775)
0810 23:00:20.426191 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:20.426252 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":73,"threads_started":1}
I20260810 23:00:20.611665 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:20.611260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.131:42197)
0810 23:00:20.611508 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:20.611561 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260810 23:00:21.289754 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:21.289361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.129:46719)
0810 23:00:21.289607 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:21.289666 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260810 23:00:21.427621 16557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:21.427237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:35109 (local address 127.29.78.130:39005)
0810 23:00:21.427479 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:21.427530 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.190:35109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260810 23:00:21.596426 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 16018
I20260810 23:00:21.604683 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 16206
I20260810 23:00:21.611490 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 16353
2026-08-10T23:00:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59341 ms)
[----------] 30 tests from SecurityITest (242799 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:21 dist-test-slave-6nmv krb5kdc[16560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:21 dist-test-slave-6nmv krb5kdc[16560](info): set up 2 sockets
Aug 10 23:00:21 dist-test-slave-6nmv krb5kdc[16560](info): commencing operation
krb5kdc: starting...
W20260810 23:00:22.677250 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:22 dist-test-slave-6nmv krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:22.796090 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43311
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:22.900185 16576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:22.900439 16576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:22.900506 16576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:22.904090 16576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:22.904170 16576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:22.904208 16576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:22.904265 16576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:22.904320 16576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:22.909147 16576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:22.910059 16576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:22.911201 16576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:00:22.919768 16576 server_base.cc:1034] running on GCE node
W20260810 23:00:22.919709 16584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:22.919734 16583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:22.919737 16586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:22.920471 16576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:22.921429 16576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:22.922601 16576 hybrid_clock.cc:648] HybridClock initialized: now 1786402822922571 us; error 39 us; skew 500 ppm
Aug 10 23:00:22 dist-test-slave-6nmv krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402822, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:22.925287 16576 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:22.926272 16576 webserver.cc:460] Webserver started at http://127.29.78.190:43647/ using document root <none> and password file <none>
I20260810 23:00:22.926802 16576 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:22.926892 16576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:22.927122 16576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:22.929075 16576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "5c0e54fa94fd4defaa02b99ee0d90709"
format_stamp: "Formatted at 2026-08-10 23:00:22 on dist-test-slave-6nmv"
server_key: "0ec15ba84ffbd18449c8f58dd0da661e"
server_key_iv: "36a9df917ac2997840046d2d2619308b"
server_key_version: "encryptionkey@0"
I20260810 23:00:22.929625 16576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "5c0e54fa94fd4defaa02b99ee0d90709"
format_stamp: "Formatted at 2026-08-10 23:00:22 on dist-test-slave-6nmv"
server_key: "0ec15ba84ffbd18449c8f58dd0da661e"
server_key_iv: "36a9df917ac2997840046d2d2619308b"
server_key_version: "encryptionkey@0"
I20260810 23:00:22.933120 16576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:00:22.935382 16594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:22.936570 16576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:22.936709 16576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "5c0e54fa94fd4defaa02b99ee0d90709"
format_stamp: "Formatted at 2026-08-10 23:00:22 on dist-test-slave-6nmv"
server_key: "0ec15ba84ffbd18449c8f58dd0da661e"
server_key_iv: "36a9df917ac2997840046d2d2619308b"
server_key_version: "encryptionkey@0"
I20260810 23:00:22.936820 16576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:22.952201 16576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:22.954757 16576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:22.954962 16576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:22.962069 16576 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43311
I20260810 23:00:22.962096 16651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43311 every 8 connection(s)
I20260810 23:00:22.962916 16576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:22.965493 16652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:22.970688 16652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709: Bootstrap starting.
I20260810 23:00:22.971678 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 16576
I20260810 23:00:22.971812 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:22.972108 30010 external_mini_cluster.cc:1428] Setting key 24eb718265d1fbae63e2dfa7faf04c34
I20260810 23:00:22.973305 16652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:22.974107 16652 log.cc:824] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:22.975931 16652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709: No bootstrap required, opened a new log
I20260810 23:00:22.979302 16652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43311 } }
I20260810 23:00:22.979542 16652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:22.979635 16652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c0e54fa94fd4defaa02b99ee0d90709, State: Initialized, Role: FOLLOWER
I20260810 23:00:22.980113 16652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43311 } }
I20260810 23:00:22.980264 16652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:22.980333 16652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:22.980424 16652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 23:00:22 dist-test-slave-6nmv krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:22.981197 16652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43311 } }
I20260810 23:00:22.981599 16652 leader_election.cc:302] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c0e54fa94fd4defaa02b99ee0d90709; no voters:
I20260810 23:00:22.981992 16652 leader_election.cc:288] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:22.982043 16658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:22.982287 16658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [term 1 LEADER]: Becoming Leader. State: Replica: 5c0e54fa94fd4defaa02b99ee0d90709, State: Running, Role: LEADER
I20260810 23:00:22.982618 16658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43311 } }
I20260810 23:00:22.983089 16652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:22.984480 16659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43311 } } }
I20260810 23:00:22.984606 16659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:22.985543 16660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c0e54fa94fd4defaa02b99ee0d90709. Latest consensus state: current_term: 1 leader_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c0e54fa94fd4defaa02b99ee0d90709" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43311 } } }
I20260810 23:00:22.986380 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:22.973208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.29.78.190:43311)
0810 23:00:22.973596 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:22.973610 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:22.973642 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:00:22.974333 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:22.974349 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:22.974666 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:22.975046 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:22.975845 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:22.976844 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:22.977686 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:22.978182 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:22.978348 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:22.981034 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:22.981038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:22.981049 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:22.981072 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:22.983052 (+ 1980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:22.983539 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:22.983544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:22.983550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:22.983682 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:22.984007 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:22.984011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:22.984013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:22.984528 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:22.984712 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:22.985135 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 23:00:22.985428 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260810 23:00:22.986797 16660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709 [sys.catalog]: This master's current role is: LEADER
W20260810 23:00:22.988301 16676 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:00:22.988374 16676 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:00:22.988482 16672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:22.989266 16672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:22.994761 16672 catalog_manager.cc:1347] Generated new cluster ID: 3ca7801ba24143b4b6d06087c166ba6f
I20260810 23:00:22.994831 16672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:23.001999 16672 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:23.002777 16672 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:23.006745 16672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709: Generated new TSK 0
I20260810 23:00:23.007345 16672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 23:00:23.037137 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43311
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:23.143733 16683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:23.143940 16683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:23.143978 16683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:23.147802 16683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:23.147853 16683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:23.147902 16683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 23:00:23.153040 16683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:23.154069 16683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:23.155045 16683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:23.164027 16692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:23.164067 16689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:23.164117 16683 server_base.cc:1034] running on GCE node
W20260810 23:00:23.164067 16690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:23.164551 16683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:23.165091 16683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:23.166242 16683 hybrid_clock.cc:648] HybridClock initialized: now 1786402823166218 us; error 39 us; skew 500 ppm
Aug 10 23:00:23 dist-test-slave-6nmv krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402823, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:23.168743 16683 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 23:00:23.169548 16683 webserver.cc:460] Webserver started at http://127.29.78.129:35089/ using document root <none> and password file <none>
I20260810 23:00:23.170106 16683 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:23.170163 16683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:23.170349 16683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:23.172178 16683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "b4aa8c5cc0334ffaa035c33db1219af3"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "76ab5d03161480f46e7d21e692fdbd4a"
server_key_iv: "4f83f464601a084b2feaa9960eea686a"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.172706 16683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "b4aa8c5cc0334ffaa035c33db1219af3"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "76ab5d03161480f46e7d21e692fdbd4a"
server_key_iv: "4f83f464601a084b2feaa9960eea686a"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.176323 16683 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:00:23.178587 16699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.179730 16683 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:23.179859 16683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "b4aa8c5cc0334ffaa035c33db1219af3"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "76ab5d03161480f46e7d21e692fdbd4a"
server_key_iv: "4f83f464601a084b2feaa9960eea686a"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.179973 16683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:23.194563 16683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:23.197172 16683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:23.197350 16683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:23.197907 16683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:23.198638 16683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:23.198704 16683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.198767 16683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:23.198817 16683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.207865 16683 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:46837
I20260810 23:00:23.207950 16823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:46837 every 8 connection(s)
I20260810 23:00:23.208725 16683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 23:00:23.212980 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 16683
I20260810 23:00:23.213111 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 23:00:23.213382 30010 external_mini_cluster.cc:1428] Setting key 5c8177293c3eaade44570bccb8d79760
Aug 10 23:00:23 dist-test-slave-6nmv krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402823, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:23.222579 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.210479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:36323 (local address 127.29.78.190:43311)
0810 23:00:23.210620 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.210623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.211403 (+ 780us) server_negotiation.cc:407] Connection header received
0810 23:00:23.212734 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.212738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.212807 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.212917 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:23.214252 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.214894 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.215987 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.216399 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.216534 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.219117 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:23.219123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:23.219129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:23.219160 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:23.220670 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.221100 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.221103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.221104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.221183 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.221513 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.221516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.221518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.221682 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:23.221786 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.222285 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.222418 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:00:23.223385 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.210835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43311 (local address 127.29.78.129:36323)
0810 23:00:23.211255 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.211294 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.212529 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.213126 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.213132 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.213542 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:23.214117 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.214136 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.215017 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.215020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.215861 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.215871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.216543 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.216546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.216766 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.217378 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:23.217400 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:23.218962 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:23.220764 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:23.220768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:23.220777 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:23.220992 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:23.221268 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:23.221271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:23.221272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:23.221410 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:23.221782 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:23.221787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:23.222059 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.222231 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.222537 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 23:00:23.224718 16824 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43311
I20260810 23:00:23.224994 16824 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:23.225499 16824 heartbeater.cc:511] Master 127.29.78.190:43311 requested a full tablet report, sending...
I20260810 23:00:23.226958 16614 ts_manager.cc:194] Registered new tserver with Master: b4aa8c5cc0334ffaa035c33db1219af3 (127.29.78.129:46837)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 23:00:23.228039 16614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:36323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 23:00:23.242877 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43311
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:23.350028 16830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:23.350245 16830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:23.350319 16830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:23.354450 16830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:23.354497 16830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:23.354592 16830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 23:00:23.359488 16830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:23.360369 16830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:23.361333 16830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:23.370214 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:23.370215 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:23.370363 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:23.370447 16830 server_base.cc:1034] running on GCE node
I20260810 23:00:23.370827 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:23.371421 16830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:23.372601 16830 hybrid_clock.cc:648] HybridClock initialized: now 1786402823372593 us; error 30 us; skew 500 ppm
Aug 10 23:00:23 dist-test-slave-6nmv krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402823, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:23.375253 16830 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 23:00:23.376084 16830 webserver.cc:460] Webserver started at http://127.29.78.130:43641/ using document root <none> and password file <none>
I20260810 23:00:23.376627 16830 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:23.376720 16830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:23.376942 16830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:23.379016 16830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "0a61265641794809b136c1b63522f21b"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "7bd69e2d7cbcceb804cea5bcfa1c0aa8"
server_key_iv: "e721c6912121605c944e7930be78d7e2"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.379587 16830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "0a61265641794809b136c1b63522f21b"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "7bd69e2d7cbcceb804cea5bcfa1c0aa8"
server_key_iv: "e721c6912121605c944e7930be78d7e2"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.383247 16830 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:00:23.386153 16845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.387362 16830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:23.387508 16830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "0a61265641794809b136c1b63522f21b"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "7bd69e2d7cbcceb804cea5bcfa1c0aa8"
server_key_iv: "e721c6912121605c944e7930be78d7e2"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.387617 16830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:23.403859 16830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:23.406656 16830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:23.406849 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:23.407394 16830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:23.408211 16830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:23.408286 16830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.408349 16830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:23.408393 16830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.417203 16830 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:39399
I20260810 23:00:23.417244 16983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:39399 every 8 connection(s)
I20260810 23:00:23.418059 16830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 23:00:23.418583 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 16830
I20260810 23:00:23.418717 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 23:00:23.419066 30010 external_mini_cluster.cc:1428] Setting key 51fcb4075696e4922ee48f96d0362082
Aug 10 23:00:23 dist-test-slave-6nmv krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402823, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 23:00:23.432381 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.419917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:32901 (local address 127.29.78.190:43311)
0810 23:00:23.420055 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.420061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.420986 (+ 925us) server_negotiation.cc:407] Connection header received
0810 23:00:23.422285 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.422291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.422351 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.422445 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:23.423529 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.424019 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.425428 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.425889 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.426032 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.428670 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:23.428676 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:23.428678 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:23.428708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:23.430421 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.430887 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.430891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.430894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.431003 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.431326 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.431328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.431329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.431506 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:23.431596 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.432159 (+ 563us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.432262 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:00:23.433291 16986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.420394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43311 (local address 127.29.78.130:32901)
0810 23:00:23.420827 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.420875 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.422095 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.422587 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.422593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.422906 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:23.423348 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.423360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.424140 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.424147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.425281 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.425291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.426037 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.426040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.426176 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.426778 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:23.426793 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:23.428501 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:23.430534 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:23.430538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:23.430548 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:23.430781 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:23.431106 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:23.431108 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:23.431109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:23.431238 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:23.431608 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:23.431613 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:23.431866 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.432105 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.432344 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260810 23:00:23.434514 16984 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43311
I20260810 23:00:23.434777 16984 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:23.435330 16984 heartbeater.cc:511] Master 127.29.78.190:43311 requested a full tablet report, sending...
I20260810 23:00:23.436580 16614 ts_manager.cc:194] Registered new tserver with Master: 0a61265641794809b136c1b63522f21b (127.29.78.130:39399)
I20260810 23:00:23.437460 16614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:32901
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 23:00:23.448721 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43311
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:23.559119 16989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:23.559352 16989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:23.559389 16989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:23.562904 16989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:23.562952 16989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:23.563004 16989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 23:00:23.567956 16989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:43311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:23.568846 16989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:23.569715 16989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:23.579334 16998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:23.579391 16995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:23.579365 16996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:23.579632 16989 server_base.cc:1034] running on GCE node
I20260810 23:00:23.580075 16989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:23.580683 16989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:23.581887 16989 hybrid_clock.cc:648] HybridClock initialized: now 1786402823581873 us; error 26 us; skew 500 ppm
Aug 10 23:00:23 dist-test-slave-6nmv krb5kdc[16560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402823, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:23.585064 16989 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 23:00:23.586033 16989 webserver.cc:460] Webserver started at http://127.29.78.131:44855/ using document root <none> and password file <none>
I20260810 23:00:23.586601 16989 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:23.586661 16989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:23.586905 16989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:23.588933 16989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "1804c0e2a5f843abb7671e3bdc3b8efa"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "7540bc692a43cb622be6ccf757ef9bed"
server_key_iv: "7acd45ab04fc8c854faf58684a1443e7"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.589494 16989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "1804c0e2a5f843abb7671e3bdc3b8efa"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "7540bc692a43cb622be6ccf757ef9bed"
server_key_iv: "7acd45ab04fc8c854faf58684a1443e7"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.593377 16989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 23:00:23.595791 17005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.597018 16989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:23.597162 16989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "1804c0e2a5f843abb7671e3bdc3b8efa"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "7540bc692a43cb622be6ccf757ef9bed"
server_key_iv: "7acd45ab04fc8c854faf58684a1443e7"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.597262 16989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:23.609349 16989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:23.612365 16989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:23.612576 16989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:23.613181 16989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:23.614045 16989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:23.614092 16989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.614130 16989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:23.614183 16989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.622889 16989 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:39183
I20260810 23:00:23.622913 17134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:39183 every 8 connection(s)
I20260810 23:00:23.623767 16989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 23:00:23.624821 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 16989
I20260810 23:00:23.624969 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 23:00:23.625242 30010 external_mini_cluster.cc:1428] Setting key 5f6a96430069e14801cce6dd7dc5b1c7
Aug 10 23:00:23 dist-test-slave-6nmv krb5kdc[16560](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402823, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:23.637548 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.625758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:60793 (local address 127.29.78.190:43311)
0810 23:00:23.625940 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.625945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.626966 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 23:00:23.628442 (+ 1476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.628445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.628499 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.628582 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:23.629736 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.630337 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.631496 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.631741 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.631854 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.634269 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:23.634277 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:23.634279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:23.634301 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:23.635563 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.636058 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.636062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.636064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.636152 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.636485 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.636490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.636491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.636673 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:23.636790 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.637272 (+ 482us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.637391 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 23:00:23.638581 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.626234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43311 (local address 127.29.78.131:60793)
0810 23:00:23.626799 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.626855 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.628213 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.628764 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.628771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.629114 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:23.629579 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.629592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.630471 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.630475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.631362 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.631371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.631892 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.631897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.632070 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.632636 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:23.632653 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:23.634090 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:23.635681 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:23.635687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:23.635700 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:23.635955 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:23.636247 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:23.636251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:23.636253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:23.636388 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:23.636803 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:23.636810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:23.637052 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.637231 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.637541 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":150,"threads_started":1}
I20260810 23:00:23.639657 17135 heartbeater.cc:347] Connected to a master server at 127.29.78.190:43311
I20260810 23:00:23.639889 17135 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:23.640437 17135 heartbeater.cc:511] Master 127.29.78.190:43311 requested a full tablet report, sending...
I20260810 23:00:23.641446 16614 ts_manager.cc:194] Registered new tserver with Master: 1804c0e2a5f843abb7671e3bdc3b8efa (127.29.78.131:39183)
I20260810 23:00:23.642076 16614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:60793
I20260810 23:00:23.651093 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:00:23.659694 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.652973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.29.78.190:43311)
0810 23:00:23.653081 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.653085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.653255 (+ 170us) server_negotiation.cc:407] Connection header received
0810 23:00:23.653351 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.653353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.653388 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.653454 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:23.654158 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.654618 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.655359 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.655553 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.655624 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.656969 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:23.656971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:23.656973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:23.656991 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:23.658268 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.658675 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.658678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.658679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.658775 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:23.659008 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:23.659010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:23.659011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:23.659208 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:23.659304 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.659488 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.659576 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:00:23.662374 16614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:00:23.665321 16614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:00:23.680287 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.674121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.29.78.130:39399)
0810 23:00:23.674469 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.674698 (+ 229us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.675218 (+ 520us) server_negotiation.cc:407] Connection header received
0810 23:00:23.675309 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.675314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.675465 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.675580 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:23.675978 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.677091 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.678472 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.679289 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.679395 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.679537 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.680042 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.680142 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":151,"threads_started":1}
I20260810 23:00:23.680290 17146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.673770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46837 (local address 127.0.0.1:33524)
0810 23:00:23.674419 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.674479 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.674596 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.675649 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.675651 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.675668 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:23.675812 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.675818 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.677145 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.677148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.678365 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.678375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.679604 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.679608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.679832 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.679889 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.679995 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.680157 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":172,"thread_start_us":119,"threads_started":1}
I20260810 23:00:23.680313 17149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.674489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39399 (local address 127.0.0.1:50280)
0810 23:00:23.674983 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.674995 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.675117 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.675610 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.675616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.675637 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:23.675818 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.675827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.677245 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.677248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.678338 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.678348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.679604 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.679608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.679838 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.679890 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.679995 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.680151 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":129,"thread_start_us":98,"threads_started":1}
I20260810 23:00:23.680356 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.673919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.29.78.129:46837)
0810 23:00:23.675257 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.675267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.675290 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:00:23.675363 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.675366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.675497 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.675654 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:23.675981 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.676978 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.678519 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.679289 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.679395 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.679537 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.680124 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.680218 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":128,"threads_started":1}
I20260810 23:00:23.680862 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.675024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.29.78.131:39183)
0810 23:00:23.675983 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.675996 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.676016 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:00:23.676087 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.676091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.676253 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.676394 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:23.676672 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.677682 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.678847 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.679512 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.679609 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.679749 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.680683 (+ 934us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.680764 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":113,"threads_started":1}
I20260810 23:00:23.681139 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.674859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:39183 (local address 127.0.0.1:36538)
0810 23:00:23.675130 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.675141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.675211 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.676398 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.676401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.676419 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:23.676540 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.676547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.677844 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.677848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.678742 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.678749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.679806 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.679809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.679942 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.679969 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.680039 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.681057 (+ 1018us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":23,"thread_start_us":132,"threads_started":1}
I20260810 23:00:23.683599 16750 tablet_service.cc:1467] Processing CreateTablet for tablet 53df0cbcea6541e28c2870370bd3829a (DEFAULT_TABLE table=test-table [id=8ed5e4bfb03b45c285da7a2623e39064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:23.684015 17060 tablet_service.cc:1467] Processing CreateTablet for tablet 53df0cbcea6541e28c2870370bd3829a (DEFAULT_TABLE table=test-table [id=8ed5e4bfb03b45c285da7a2623e39064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:23.683992 16908 tablet_service.cc:1467] Processing CreateTablet for tablet 53df0cbcea6541e28c2870370bd3829a (DEFAULT_TABLE table=test-table [id=8ed5e4bfb03b45c285da7a2623e39064]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:23.684746 16750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53df0cbcea6541e28c2870370bd3829a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:23.684945 17060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53df0cbcea6541e28c2870370bd3829a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:23.684980 16908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53df0cbcea6541e28c2870370bd3829a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:23.691641 17152 tablet_bootstrap.cc:492] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Bootstrap starting.
I20260810 23:00:23.692209 17154 tablet_bootstrap.cc:492] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Bootstrap starting.
I20260810 23:00:23.692379 17153 tablet_bootstrap.cc:492] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: Bootstrap starting.
I20260810 23:00:23.694240 17153 tablet_bootstrap.cc:654] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:23.694237 17154 tablet_bootstrap.cc:654] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:23.694237 17152 tablet_bootstrap.cc:654] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:23.695317 17153 log.cc:824] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:23.695601 17152 log.cc:824] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:23.695601 17154 log.cc:824] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:23.697456 17153 tablet_bootstrap.cc:492] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: No bootstrap required, opened a new log
I20260810 23:00:23.697649 17153 ts_tablet_manager.cc:1397] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:00:23.697686 17152 tablet_bootstrap.cc:492] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: No bootstrap required, opened a new log
I20260810 23:00:23.697718 17154 tablet_bootstrap.cc:492] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: No bootstrap required, opened a new log
I20260810 23:00:23.697903 17154 ts_tablet_manager.cc:1397] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:00:23.697935 17152 ts_tablet_manager.cc:1397] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:00:23.700662 17152 raft_consensus.cc:348] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.700964 17152 raft_consensus.cc:374] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:23.701045 17152 raft_consensus.cc:729] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a61265641794809b136c1b63522f21b, State: Initialized, Role: FOLLOWER
I20260810 23:00:23.701396 17154 raft_consensus.cc:348] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.701396 17153 raft_consensus.cc:348] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.701534 17152 consensus_queue.cc:260] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.701646 17154 raft_consensus.cc:374] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:23.701646 17153 raft_consensus.cc:374] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:23.701704 17153 raft_consensus.cc:729] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4aa8c5cc0334ffaa035c33db1219af3, State: Initialized, Role: FOLLOWER
I20260810 23:00:23.701742 17154 raft_consensus.cc:729] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1804c0e2a5f843abb7671e3bdc3b8efa, State: Initialized, Role: FOLLOWER
I20260810 23:00:23.702207 16984 heartbeater.cc:503] Master 127.29.78.190:43311 was elected leader, sending a full tablet report...
I20260810 23:00:23.702220 17153 consensus_queue.cc:260] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.702220 17154 consensus_queue.cc:260] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.702857 16824 heartbeater.cc:503] Master 127.29.78.190:43311 was elected leader, sending a full tablet report...
I20260810 23:00:23.703172 17153 ts_tablet_manager.cc:1428] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260810 23:00:23.703271 17135 heartbeater.cc:503] Master 127.29.78.190:43311 was elected leader, sending a full tablet report...
I20260810 23:00:23.704107 17154 ts_tablet_manager.cc:1428] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:00:23.704905 17152 ts_tablet_manager.cc:1428] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
W20260810 23:00:23.709748 16826 tablet.cc:2369] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:23.723330 17159 raft_consensus.cc:482] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:23.723472 17159 raft_consensus.cc:504] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.724606 17159 leader_election.cc:288] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a61265641794809b136c1b63522f21b (127.29.78.130:39399), 1804c0e2a5f843abb7671e3bdc3b8efa (127.29.78.131:39183)
I20260810 23:00:23.728626 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.724796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:39399 (local address 127.29.78.129:41413)
0810 23:00:23.725038 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.725052 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.725146 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.725425 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.725429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.725450 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:23.725576 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.725583 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.726487 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.726490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.727429 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.727438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.728133 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.728139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.728313 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.728332 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.728392 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.728448 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":31,"thread_start_us":106,"threads_started":1}
I20260810 23:00:23.728641 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.724876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:41413 (local address 127.29.78.130:39399)
0810 23:00:23.725019 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.725022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.725117 (+ 95us) server_negotiation.cc:407] Connection header received
0810 23:00:23.725228 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.725230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.725267 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.725378 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:23.725699 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.726377 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.727560 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.728000 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.728104 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.728190 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.728436 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.728526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:00:23.728744 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.725411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:48293 (local address 127.29.78.131:39183)
0810 23:00:23.725526 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.725532 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.725548 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:00:23.725632 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.725638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.725689 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.725752 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:23.726082 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.726651 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.727667 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.728083 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.728214 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.728265 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.728541 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.728612 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:00:23.728796 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.725159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:39183 (local address 127.29.78.129:48293)
0810 23:00:23.725448 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.725473 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:23.725560 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:23.725783 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:23.725786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:23.725803 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:23.725963 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.725971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.726762 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.726764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.727549 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:23.727559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.728233 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.728236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:23.728395 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.728424 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:00:23.728511 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:00:23.728578 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":31,"thread_start_us":104,"threads_started":1}
I20260810 23:00:23.729413 16932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53df0cbcea6541e28c2870370bd3829a" candidate_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a61265641794809b136c1b63522f21b" is_pre_election: true
I20260810 23:00:23.729739 16932 raft_consensus.cc:2454] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4aa8c5cc0334ffaa035c33db1219af3 in term 0.
I20260810 23:00:23.729709 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53df0cbcea6541e28c2870370bd3829a" candidate_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" is_pre_election: true
I20260810 23:00:23.729990 17082 raft_consensus.cc:2454] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4aa8c5cc0334ffaa035c33db1219af3 in term 0.
I20260810 23:00:23.730214 16701 leader_election.cc:302] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a61265641794809b136c1b63522f21b, b4aa8c5cc0334ffaa035c33db1219af3; no voters:
I20260810 23:00:23.730473 17159 raft_consensus.cc:2781] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:00:23.730547 17159 raft_consensus.cc:482] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:00:23.730576 17159 raft_consensus.cc:3037] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:23.731729 17159 raft_consensus.cc:504] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.732067 17159 leader_election.cc:288] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [CANDIDATE]: Term 1 election: Requested vote from peers 0a61265641794809b136c1b63522f21b (127.29.78.130:39399), 1804c0e2a5f843abb7671e3bdc3b8efa (127.29.78.131:39183)
I20260810 23:00:23.732435 16932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53df0cbcea6541e28c2870370bd3829a" candidate_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a61265641794809b136c1b63522f21b"
I20260810 23:00:23.732496 17082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53df0cbcea6541e28c2870370bd3829a" candidate_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa"
I20260810 23:00:23.732553 16932 raft_consensus.cc:3037] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:23.732645 17082 raft_consensus.cc:3037] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:23.733516 16932 raft_consensus.cc:2454] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4aa8c5cc0334ffaa035c33db1219af3 in term 1.
I20260810 23:00:23.733727 17082 raft_consensus.cc:2454] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4aa8c5cc0334ffaa035c33db1219af3 in term 1.
I20260810 23:00:23.733860 16701 leader_election.cc:302] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a61265641794809b136c1b63522f21b, b4aa8c5cc0334ffaa035c33db1219af3; no voters:
I20260810 23:00:23.734076 17159 raft_consensus.cc:2781] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:23.734304 17159 raft_consensus.cc:686] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 1 LEADER]: Becoming Leader. State: Replica: b4aa8c5cc0334ffaa035c33db1219af3, State: Running, Role: LEADER
I20260810 23:00:23.734689 17159 consensus_queue.cc:237] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } }
I20260810 23:00:23.737773 16614 catalog_manager.cc:5495] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4aa8c5cc0334ffaa035c33db1219af3 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4aa8c5cc0334ffaa035c33db1219af3" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20260810 23:00:23.760210 17147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.757061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.29.78.129:46837)
0810 23:00:23.757194 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:23.757197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:23.757210 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:00:23.757285 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:23.757287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:23.757342 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:23.757449 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:00:23.757685 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.758184 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.758968 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:23.759164 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:23.759235 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:23.759504 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:00:23.759904 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:00:23.759992 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:23.760055 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 23:00:23.760103 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:00:23.766079 16932 raft_consensus.cc:1261] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 1 FOLLOWER]: Refusing update from remote peer b4aa8c5cc0334ffaa035c33db1219af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:23.766079 17082 raft_consensus.cc:1261] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 1 FOLLOWER]: Refusing update from remote peer b4aa8c5cc0334ffaa035c33db1219af3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:23.766790 17159 consensus_queue.cc:1035] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1804c0e2a5f843abb7671e3bdc3b8efa" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:23.767079 17164 consensus_queue.cc:1035] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a61265641794809b136c1b63522f21b" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 39399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:23.774426 17167 mvcc.cc:205] Tried to move back new op lower bound from 7317105966138421248 to 7317105966020182016. Current Snapshot: MvccSnapshot[applied={T|T < 7317105966138421248}]
I20260810 23:00:23.775082 17168 mvcc.cc:205] Tried to move back new op lower bound from 7317105966138421248 to 7317105966020182016. Current Snapshot: MvccSnapshot[applied={T|T < 7317105966138421248}]
I20260810 23:00:23.783550 16611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:23.783726 16611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:23.786342 16611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c0e54fa94fd4defaa02b99ee0d90709: Sending DeleteTablet for 3 replicas of tablet 53df0cbcea6541e28c2870370bd3829a
I20260810 23:00:23.787096 16750 tablet_service.cc:1514] Processing DeleteTablet for tablet 53df0cbcea6541e28c2870370bd3829a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:23 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:33524
I20260810 23:00:23.787303 16908 tablet_service.cc:1514] Processing DeleteTablet for tablet 53df0cbcea6541e28c2870370bd3829a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:23 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:50280
I20260810 23:00:23.787616 17060 tablet_service.cc:1514] Processing DeleteTablet for tablet 53df0cbcea6541e28c2870370bd3829a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:23 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:36538
I20260810 23:00:23.787801 17178 tablet_replica.cc:330] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: stopping tablet replica
I20260810 23:00:23.788106 17179 tablet_replica.cc:330] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: stopping tablet replica
I20260810 23:00:23.788317 17177 tablet_replica.cc:330] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3: stopping tablet replica
I20260810 23:00:23.788434 17179 raft_consensus.cc:2229] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:23.788686 17177 raft_consensus.cc:2229] T 53df0cbcea6541e28c2870370bd3829a P b4aa8c5cc0334ffaa035c33db1219af3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:00:23.788733 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 16683
I20260810 23:00:23.791836 17178 raft_consensus.cc:2229] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:23.792210 17178 raft_consensus.cc:2258] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:23.794179 17178 ts_tablet_manager.cc:1905] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:23.795949 17179 raft_consensus.cc:2258] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:23.797435 17179 ts_tablet_manager.cc:1905] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:23.797714 17178 ts_tablet_manager.cc:1918] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:23.797842 17178 log.cc:1196] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/53df0cbcea6541e28c2870370bd3829a
I20260810 23:00:23.798233 17178 ts_tablet_manager.cc:1939] T 53df0cbcea6541e28c2870370bd3829a P 0a61265641794809b136c1b63522f21b: Deleting consensus metadata
I20260810 23:00:23.799674 16596 catalog_manager.cc:4841] TS 0a61265641794809b136c1b63522f21b (127.29.78.130:39399): tablet 53df0cbcea6541e28c2870370bd3829a (table test-table [id=8ed5e4bfb03b45c285da7a2623e39064]) successfully deleted
I20260810 23:00:23.799762 17179 ts_tablet_manager.cc:1918] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:23.799888 17179 log.cc:1196] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/53df0cbcea6541e28c2870370bd3829a
I20260810 23:00:23.800230 17179 ts_tablet_manager.cc:1939] T 53df0cbcea6541e28c2870370bd3829a P 1804c0e2a5f843abb7671e3bdc3b8efa: Deleting consensus metadata
I20260810 23:00:23.801226 16596 catalog_manager.cc:4841] TS 1804c0e2a5f843abb7671e3bdc3b8efa (127.29.78.131:39183): tablet 53df0cbcea6541e28c2870370bd3829a (table test-table [id=8ed5e4bfb03b45c285da7a2623e39064]) successfully deleted
W20260810 23:00:23.803267 16598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:46837 (error 108)
I20260810 23:00:23.804028 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 16830
I20260810 23:00:23.804323 17150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:23.803996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:46837 (local address 127.0.0.1:33534)
0810 23:00:23.804159 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:23.804233 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46837: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:00:23.809499 16598 catalog_manager.cc:4584] TS b4aa8c5cc0334ffaa035c33db1219af3 (127.29.78.129:46837): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 53df0cbcea6541e28c2870370bd3829a: Network error: Client connection negotiation failed: client connection to 127.29.78.129:46837: connect: Connection refused (error 111)
I20260810 23:00:23.817036 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 16989
I20260810 23:00:23.824535 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 16576
2026-08-10T23:00:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2211 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T23:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:23Z Disabled control of system clock
I20260810 23:00:23.856053 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38825
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:38825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:00:23.964648 17186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:23.964891 17186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:23.964941 17186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:23.968653 17186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:23.968703 17186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:23.968719 17186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:23.968734 17186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:23.968748 17186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:23.972887 17186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:38825
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:23.973752 17186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:23.975054 17186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:23.983022 17191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:23.983006 17194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:23.982997 17192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:23.983047 17186 server_base.cc:1034] running on GCE node
I20260810 23:00:23.983762 17186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:23.984714 17186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:23.985879 17186 hybrid_clock.cc:648] HybridClock initialized: now 1786402823985865 us; error 29 us; skew 500 ppm
I20260810 23:00:23.987593 17186 webserver.cc:460] Webserver started at http://127.29.78.190:35601/ using document root <none> and password file <none>
I20260810 23:00:23.988124 17186 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:23.988209 17186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:23.988399 17186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:23.990310 17186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "ec19ec7c662540c88b753e7c6a95ef56"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "3e4e77a5f207e047439a571cb747cbb0"
server_key_iv: "b7a3dc92611eb824d827aae9e133534f"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.990868 17186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "ec19ec7c662540c88b753e7c6a95ef56"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "3e4e77a5f207e047439a571cb747cbb0"
server_key_iv: "b7a3dc92611eb824d827aae9e133534f"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.994228 17186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:00:23.996325 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:23.997471 17186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:23.997608 17186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "ec19ec7c662540c88b753e7c6a95ef56"
format_stamp: "Formatted at 2026-08-10 23:00:23 on dist-test-slave-6nmv"
server_key: "3e4e77a5f207e047439a571cb747cbb0"
server_key_iv: "b7a3dc92611eb824d827aae9e133534f"
server_key_version: "encryptionkey@0"
I20260810 23:00:23.997711 17186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:24.009459 17186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:24.010061 17186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:24.010241 17186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:24.017185 17186 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:38825
I20260810 23:00:24.017191 17256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:38825 every 8 connection(s)
I20260810 23:00:24.018086 17186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:24.021021 17257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:24.021415 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 17186
I20260810 23:00:24.021552 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:24.021852 30010 external_mini_cluster.cc:1428] Setting key 14645d8fd82dca6d69b07d369d6de19a
I20260810 23:00:24.026623 17257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56: Bootstrap starting.
I20260810 23:00:24.028852 17257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:24.029556 17257 log.cc:824] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:24.030792 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.023167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40200 (local address 127.29.78.190:38825)
0810 23:00:24.023709 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.023721 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.023752 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:00:24.024323 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.024337 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.024767 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.025071 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:24.025442 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.026434 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.027373 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.027825 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.027991 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.028985 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:24.028990 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:24.029002 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:24.029354 (+ 352us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:24.029401 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:24.029418 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:24.029525 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.029670 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.029959 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260810 23:00:24.031679 17257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56: No bootstrap required, opened a new log
I20260810 23:00:24.035260 17257 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19ec7c662540c88b753e7c6a95ef56" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38825 } }
I20260810 23:00:24.035444 17257 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:24.035493 17257 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec19ec7c662540c88b753e7c6a95ef56, State: Initialized, Role: FOLLOWER
I20260810 23:00:24.035799 17257 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19ec7c662540c88b753e7c6a95ef56" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38825 } }
I20260810 23:00:24.035919 17257 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:24.035974 17257 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:24.036042 17257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:24.036818 17257 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19ec7c662540c88b753e7c6a95ef56" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38825 } }
I20260810 23:00:24.037093 17257 leader_election.cc:302] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec19ec7c662540c88b753e7c6a95ef56; no voters:
I20260810 23:00:24.037386 17257 leader_election.cc:288] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:24.037482 17263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:24.037755 17263 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [term 1 LEADER]: Becoming Leader. State: Replica: ec19ec7c662540c88b753e7c6a95ef56, State: Running, Role: LEADER
I20260810 23:00:24.038056 17263 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19ec7c662540c88b753e7c6a95ef56" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38825 } }
I20260810 23:00:24.038763 17257 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:24.039767 17265 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec19ec7c662540c88b753e7c6a95ef56. Latest consensus state: current_term: 1 leader_uuid: "ec19ec7c662540c88b753e7c6a95ef56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19ec7c662540c88b753e7c6a95ef56" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38825 } } }
I20260810 23:00:24.039922 17265 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:24.040298 17272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:24.040901 17264 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec19ec7c662540c88b753e7c6a95ef56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec19ec7c662540c88b753e7c6a95ef56" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 38825 } } }
I20260810 23:00:24.041052 17264 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:24.043303 17272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:24.049955 17272 catalog_manager.cc:1347] Generated new cluster ID: 004128a0240f4c66971480d852bad551
I20260810 23:00:24.050036 17272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:24.073544 17272 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:24.074527 17272 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:24.084661 17272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56: Generated new TSK 0
I20260810 23:00:24.085362 17272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:24.093674 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38825
--builtin_ntp_servers=127.29.78.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:00:24.200174 17282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:24.200384 17282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:24.200416 17282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:24.204327 17282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:24.204373 17282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:24.204420 17282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 23:00:24.208611 17282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:24.209535 17282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:24.210807 17282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:24.219864 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:24.219877 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:24.220008 17282 server_base.cc:1034] running on GCE node
W20260810 23:00:24.219942 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:24.220566 17282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:24.221129 17282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:24.222296 17282 hybrid_clock.cc:648] HybridClock initialized: now 1786402824222277 us; error 30 us; skew 500 ppm
I20260810 23:00:24.224014 17282 webserver.cc:460] Webserver started at http://127.29.78.129:42497/ using document root <none> and password file <none>
I20260810 23:00:24.224550 17282 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:24.224609 17282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:24.224848 17282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:24.226908 17282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "6d6ad1f4015724e3d3f611ba2458b634"
server_key_iv: "91a69037c5acd275789d6ed83e183c0c"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.227485 17282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "6d6ad1f4015724e3d3f611ba2458b634"
server_key_iv: "91a69037c5acd275789d6ed83e183c0c"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.231015 17282 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:00:24.233229 17297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.234385 17282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:24.234512 17282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "6d6ad1f4015724e3d3f611ba2458b634"
server_key_iv: "91a69037c5acd275789d6ed83e183c0c"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.234616 17282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:24.243551 17282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:24.244107 17282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:24.244287 17282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:24.244810 17282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:24.245675 17282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:24.245743 17282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.245824 17282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:24.245867 17282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.254585 17282 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:45807
I20260810 23:00:24.254630 17428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:45807 every 8 connection(s)
I20260810 23:00:24.255386 17282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 23:00:24.260025 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 17282
I20260810 23:00:24.260142 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 23:00:24.260381 30010 external_mini_cluster.cc:1428] Setting key 4740fbde2b7d0ec9f9dc3b900e729c1e
I20260810 23:00:24.262348 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38825
--builtin_ntp_servers=127.29.78.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:00:24.264657 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.257078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33995 (local address 127.29.78.190:38825)
0810 23:00:24.257240 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.257244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.258213 (+ 969us) server_negotiation.cc:407] Connection header received
0810 23:00:24.259431 (+ 1218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.259433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.259498 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.259612 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:24.260524 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.261142 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.262642 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.262848 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.262930 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.264133 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:24.264138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:24.264142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:24.264157 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:24.264181 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:24.264193 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:24.264271 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.264409 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.264483 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 23:00:24.265287 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.257447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38825 (local address 127.29.78.129:33995)
0810 23:00:24.258079 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:24.258113 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:24.259243 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:24.259765 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:24.259772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:24.259844 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:24.260363 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.260377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.261284 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.261287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.262527 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.262541 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.263005 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.263011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.263227 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.263819 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:24.263824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:24.263874 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:24.263955 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:24.264284 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:24.264291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:24.264295 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:00:24.264381 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:00:24.264571 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260810 23:00:24.266486 17429 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38825
I20260810 23:00:24.266718 17429 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:24.267323 17429 heartbeater.cc:511] Master 127.29.78.190:38825 requested a full tablet report, sending...
I20260810 23:00:24.268702 17217 ts_manager.cc:194] Registered new tserver with Master: 4898bdd8ed5d4d5d8b52ab89dd6fe19d (127.29.78.129:45807)
I20260810 23:00:24.269685 17217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:33995
W20260810 23:00:24.367897 17432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:24.368108 17432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:24.368142 17432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:24.372655 17432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:24.372704 17432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:24.372789 17432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 23:00:24.377337 17432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:24.378362 17432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:24.379670 17432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:00:24.388942 17432 server_base.cc:1034] running on GCE node
W20260810 23:00:24.388891 17446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:24.388885 17438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:24.388902 17443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:24.389488 17432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:24.390125 17432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:24.391306 17432 hybrid_clock.cc:648] HybridClock initialized: now 1786402824391298 us; error 39 us; skew 500 ppm
I20260810 23:00:24.393172 17432 webserver.cc:460] Webserver started at http://127.29.78.130:42015/ using document root <none> and password file <none>
I20260810 23:00:24.393664 17432 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:24.393720 17432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:24.393939 17432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:24.395905 17432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "013d3b86a63148f4876c2bfe44599056"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "fd010d96b46addd38c689f5f4992da63"
server_key_iv: "f8277bc9c722615d1aff4095c54f9567"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.396437 17432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "013d3b86a63148f4876c2bfe44599056"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "fd010d96b46addd38c689f5f4992da63"
server_key_iv: "f8277bc9c722615d1aff4095c54f9567"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.400058 17432 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:00:24.402599 17453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.403738 17432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:24.403864 17432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "013d3b86a63148f4876c2bfe44599056"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "fd010d96b46addd38c689f5f4992da63"
server_key_iv: "f8277bc9c722615d1aff4095c54f9567"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.403978 17432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:24.423009 17432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:24.423636 17432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:24.423821 17432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:24.424350 17432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:24.425248 17432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:24.425316 17432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.425395 17432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:24.425431 17432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.434229 17432 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:44043
I20260810 23:00:24.434265 17582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:44043 every 8 connection(s)
I20260810 23:00:24.435081 17432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 23:00:24.439776 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 17432
I20260810 23:00:24.439903 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 23:00:24.440225 30010 external_mini_cluster.cc:1428] Setting key d72b27bc9e40f7f9a642b57563b8f049
I20260810 23:00:24.442535 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38825
--builtin_ntp_servers=127.29.78.148:35211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:00:24.445343 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.436816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49013 (local address 127.29.78.190:38825)
0810 23:00:24.437004 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.437009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.437983 (+ 974us) server_negotiation.cc:407] Connection header received
0810 23:00:24.439376 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.439379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.439427 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.439514 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:24.440690 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.441484 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.443258 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.443472 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.443554 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.444761 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:24.444766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:24.444770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:24.444785 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:24.444826 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:24.444838 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:24.444920 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.445063 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.445142 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 23:00:24.446151 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.437198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38825 (local address 127.29.78.130:49013)
0810 23:00:24.437786 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:24.437872 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:24.439183 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:24.439724 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:24.439733 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:24.439841 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:24.440512 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.440532 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.441672 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.441678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.443135 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.443150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.443634 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.443640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.443876 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.444516 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:24.444517 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:24.444554 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:24.444611 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:24.444924 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:24.444931 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:24.444935 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:00:24.445033 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:00:24.445258 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":116,"threads_started":1}
I20260810 23:00:24.447279 17583 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38825
I20260810 23:00:24.447510 17583 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:24.447990 17583 heartbeater.cc:511] Master 127.29.78.190:38825 requested a full tablet report, sending...
I20260810 23:00:24.449011 17217 ts_manager.cc:194] Registered new tserver with Master: 013d3b86a63148f4876c2bfe44599056 (127.29.78.130:44043)
I20260810 23:00:24.449541 17217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:49013
W20260810 23:00:24.551147 17586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:24.551429 17586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:24.551519 17586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:24.555922 17586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:24.555994 17586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:24.556047 17586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 23:00:24.560398 17586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:35211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:38825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:24.561239 17586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:24.562539 17586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:24.571550 17594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:24.571614 17586 server_base.cc:1034] running on GCE node
W20260810 23:00:24.571538 17591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:24.571568 17592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:24.572237 17586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:24.572827 17586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:24.573987 17586 hybrid_clock.cc:648] HybridClock initialized: now 1786402824573958 us; error 39 us; skew 500 ppm
I20260810 23:00:24.575668 17586 webserver.cc:460] Webserver started at http://127.29.78.131:42841/ using document root <none> and password file <none>
I20260810 23:00:24.576231 17586 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:24.576292 17586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:24.576542 17586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:24.578495 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "37e4ab3f4cfc48388b80de324dbacb4e"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "c3d7a2afe70b809e8cee714940f54a6d"
server_key_iv: "7a16c440ca045c0ac6ec984389f4de6d"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.579010 17586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "37e4ab3f4cfc48388b80de324dbacb4e"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "c3d7a2afe70b809e8cee714940f54a6d"
server_key_iv: "7a16c440ca045c0ac6ec984389f4de6d"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.582407 17586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:00:24.584610 17600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.585785 17586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:24.585988 17586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "37e4ab3f4cfc48388b80de324dbacb4e"
format_stamp: "Formatted at 2026-08-10 23:00:24 on dist-test-slave-6nmv"
server_key: "c3d7a2afe70b809e8cee714940f54a6d"
server_key_iv: "7a16c440ca045c0ac6ec984389f4de6d"
server_key_version: "encryptionkey@0"
I20260810 23:00:24.586090 17586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:24.597259 17586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:24.597872 17586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:24.598049 17586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:24.598573 17586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:24.599395 17586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:24.599440 17586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.599520 17586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:24.599557 17586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:24.610464 17586 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:38469
I20260810 23:00:24.610528 17727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:38469 every 8 connection(s)
I20260810 23:00:24.611310 17586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 23:00:24.619860 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 17586
I20260810 23:00:24.619995 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 23:00:24.620251 30010 external_mini_cluster.cc:1428] Setting key e9fd8885cd21aab4a6c45b636adf6047
I20260810 23:00:24.620288 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.613146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:52379 (local address 127.29.78.190:38825)
0810 23:00:24.613278 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.613282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.614303 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 23:00:24.615766 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.615770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.615827 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.615935 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:24.616900 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.617449 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.618563 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.618771 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.618878 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.619810 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:24.619813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:24.619816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:24.619834 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:24.619849 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:24.619856 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:24.619930 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.620096 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.620154 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:00:24.621284 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.613515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:38825 (local address 127.29.78.131:52379)
0810 23:00:24.614166 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:24.614205 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:24.615550 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:24.616120 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:24.616126 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:24.616206 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:24.616756 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.616771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.617580 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.617584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.618441 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.618450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.618913 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.618917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.619068 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.619578 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:24.619580 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:24.619614 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:24.619667 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:24.619931 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:24.619942 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:24.619949 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:00:24.620095 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 23:00:24.620347 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20260810 23:00:24.622303 17728 heartbeater.cc:347] Connected to a master server at 127.29.78.190:38825
I20260810 23:00:24.622519 17728 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:24.622972 17728 heartbeater.cc:511] Master 127.29.78.190:38825 requested a full tablet report, sending...
I20260810 23:00:24.623848 17217 ts_manager.cc:194] Registered new tserver with Master: 37e4ab3f4cfc48388b80de324dbacb4e (127.29.78.131:38469)
I20260810 23:00:24.624382 17217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:52379
I20260810 23:00:24.633766 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:00:24.638978 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.635523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.29.78.190:38825)
0810 23:00:24.635648 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.635653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.635869 (+ 216us) server_negotiation.cc:407] Connection header received
0810 23:00:24.635982 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.635984 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.636020 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.636081 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:24.636282 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.636744 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.637546 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.637763 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.637909 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.638570 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:24.638573 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:24.638575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:24.638585 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:24.638602 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:24.638609 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:24.638683 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.638805 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.638851 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:00:24.641523 17217 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:00:24.644254 17217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:00:24.657856 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.652597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52236 (local address 127.29.78.129:45807)
0810 23:00:24.652974 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.652985 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.653517 (+ 532us) server_negotiation.cc:407] Connection header received
0810 23:00:24.653582 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.653585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.653745 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.653898 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:24.654225 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.654888 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.656643 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.657180 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.657311 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.657490 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.657586 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.657671 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":88,"threads_started":1}
I20260810 23:00:24.657856 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.652444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:45807 (local address 127.0.0.1:52236)
0810 23:00:24.653286 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:24.653298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:24.653387 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:24.654038 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:24.654041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:24.654056 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:24.654124 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.654127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.655010 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.655013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.656527 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.656535 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.657312 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.657315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.657417 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.657451 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:00:24.657562 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 23:00:24.657671 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"mutex_wait_us":48,"thread_start_us":83,"threads_started":1}
I20260810 23:00:24.657882 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.651951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:38469 (local address 127.0.0.1:40550)
0810 23:00:24.652808 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:24.652853 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:24.652978 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:24.653899 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:24.653902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:24.653918 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:24.654038 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.654045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.655333 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.655338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.656413 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.656422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.657468 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.657471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.657553 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.657573 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:00:24.657623 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 23:00:24.657685 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":5,"thread_start_us":64,"threads_started":1}
I20260810 23:00:24.658231 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.652077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44043 (local address 127.0.0.1:33636)
0810 23:00:24.653155 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:24.653164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:24.653256 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:24.653753 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:24.653760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:24.653779 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:24.653928 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.653935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.655247 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.655249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.656389 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:24.656399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.657645 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.657648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:24.657721 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.657741 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:00:24.657802 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:00:24.657874 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260810 23:00:24.658236 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.652698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.29.78.130:44043)
0810 23:00:24.653026 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.653037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.653252 (+ 215us) server_negotiation.cc:407] Connection header received
0810 23:00:24.653359 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.653366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.653595 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.653736 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:24.654056 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.655138 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.656521 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.657316 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.657424 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.657590 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.657982 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.658090 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":140,"threads_started":1}
I20260810 23:00:24.658773 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:24.652056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.29.78.131:38469)
0810 23:00:24.653508 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:24.653518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:24.653537 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:00:24.653596 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:24.653600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:24.653744 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:24.654178 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:24.654290 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.655254 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.656575 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:24.657316 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:24.658390 (+ 1074us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:24.658508 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:24.658543 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 23:00:24.658599 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1344,"thread_start_us":141,"threads_started":1}
I20260810 23:00:24.660990 17647 tablet_service.cc:1467] Processing CreateTablet for tablet 380676e85fee4cbebaf30826036f82be (DEFAULT_TABLE table=test-table [id=3615eebf351c466ca48d9a150f482e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:24.661015 17355 tablet_service.cc:1467] Processing CreateTablet for tablet 380676e85fee4cbebaf30826036f82be (DEFAULT_TABLE table=test-table [id=3615eebf351c466ca48d9a150f482e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:24.661015 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 380676e85fee4cbebaf30826036f82be (DEFAULT_TABLE table=test-table [id=3615eebf351c466ca48d9a150f482e7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:24.661959 17647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380676e85fee4cbebaf30826036f82be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:24.662011 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380676e85fee4cbebaf30826036f82be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:24.662011 17355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380676e85fee4cbebaf30826036f82be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:24.668408 17745 tablet_bootstrap.cc:492] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Bootstrap starting.
I20260810 23:00:24.669595 17746 tablet_bootstrap.cc:492] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Bootstrap starting.
I20260810 23:00:24.670143 17747 tablet_bootstrap.cc:492] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Bootstrap starting.
I20260810 23:00:24.670655 17745 tablet_bootstrap.cc:654] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:24.671502 17745 log.cc:824] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:24.671691 17746 tablet_bootstrap.cc:654] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:24.672503 17746 log.cc:824] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:24.672736 17747 tablet_bootstrap.cc:654] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:24.673588 17747 log.cc:824] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:24.674607 17746 tablet_bootstrap.cc:492] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: No bootstrap required, opened a new log
I20260810 23:00:24.674798 17746 ts_tablet_manager.cc:1397] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:00:24.674870 17745 tablet_bootstrap.cc:492] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: No bootstrap required, opened a new log
I20260810 23:00:24.675083 17745 ts_tablet_manager.cc:1397] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 23:00:24.675400 17747 tablet_bootstrap.cc:492] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: No bootstrap required, opened a new log
I20260810 23:00:24.675580 17747 ts_tablet_manager.cc:1397] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20260810 23:00:24.678232 17745 raft_consensus.cc:348] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:24.678462 17745 raft_consensus.cc:374] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:24.678517 17745 raft_consensus.cc:729] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e4ab3f4cfc48388b80de324dbacb4e, State: Initialized, Role: FOLLOWER
I20260810 23:00:24.678906 17745 consensus_queue.cc:260] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:24.679008 17746 raft_consensus.cc:348] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:24.679242 17746 raft_consensus.cc:374] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:24.679306 17746 raft_consensus.cc:729] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4898bdd8ed5d4d5d8b52ab89dd6fe19d, State: Initialized, Role: FOLLOWER
I20260810 23:00:24.679616 17728 heartbeater.cc:503] Master 127.29.78.190:38825 was elected leader, sending a full tablet report...
I20260810 23:00:24.679524 17747 raft_consensus.cc:348] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:24.679731 17747 raft_consensus.cc:374] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:24.679778 17747 raft_consensus.cc:729] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 013d3b86a63148f4876c2bfe44599056, State: Initialized, Role: FOLLOWER
I20260810 23:00:24.679728 17746 consensus_queue.cc:260] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:24.679808 17745 ts_tablet_manager.cc:1428] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:00:24.680147 17747 consensus_queue.cc:260] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:24.680548 17746 ts_tablet_manager.cc:1428] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 23:00:24.680832 17429 heartbeater.cc:503] Master 127.29.78.190:38825 was elected leader, sending a full tablet report...
I20260810 23:00:24.681360 17583 heartbeater.cc:503] Master 127.29.78.190:38825 was elected leader, sending a full tablet report...
I20260810 23:00:24.681747 17747 ts_tablet_manager.cc:1428] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260810 23:00:24.686013 17584 tablet.cc:2369] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:00:24.756567 17430 tablet.cc:2369] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:00:24.862403 17729 tablet.cc:2369] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:25.058221 17752 raft_consensus.cc:482] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:25.058427 17752 raft_consensus.cc:504] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:25.059578 17752 leader_election.cc:288] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37e4ab3f4cfc48388b80de324dbacb4e (127.29.78.131:38469), 013d3b86a63148f4876c2bfe44599056 (127.29.78.130:44043)
I20260810 23:00:25.063189 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:25.059920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:33593 (local address 127.29.78.131:38469)
0810 23:00:25.060051 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:25.060056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:25.060167 (+ 111us) server_negotiation.cc:407] Connection header received
0810 23:00:25.060299 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:25.060302 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:25.060347 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:25.060430 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:25.060705 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.061202 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.062291 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.062660 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.062784 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:25.062832 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:25.063017 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 23:00:25.063079 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:00:25.063205 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:25.059778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:38469 (local address 127.29.78.129:33593)
0810 23:00:25.060038 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:25.060053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:25.060169 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:25.060443 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:25.060445 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:25.060462 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:25.060592 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:25.060597 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.061332 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.061336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:25.062170 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:25.062178 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.062778 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.062782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:25.062874 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:25.062899 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:00:25.062996 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:00:25.063061 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":16,"thread_start_us":88,"threads_started":1}
I20260810 23:00:25.063331 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:25.060083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:38161 (local address 127.29.78.130:44043)
0810 23:00:25.060203 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:25.060210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:25.060306 (+ 96us) server_negotiation.cc:407] Connection header received
0810 23:00:25.060459 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:25.060467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:25.060510 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:25.060588 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:25.060863 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.061382 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.062456 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.062797 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.062914 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:25.062965 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:25.063125 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 23:00:25.063188 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:00:25.063388 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:25.059856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44043 (local address 127.29.78.129:38161)
0810 23:00:25.060203 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:25.060217 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:25.060345 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:25.060599 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:25.060602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:25.060617 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:25.060744 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:25.060749 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.061504 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.061508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:25.062335 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:25.062343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.062909 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.062913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:25.063003 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:25.063027 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 23:00:25.063108 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:00:25.063169 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":19,"thread_start_us":107,"threads_started":1}
I20260810 23:00:25.063980 17668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380676e85fee4cbebaf30826036f82be" candidate_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" is_pre_election: true
I20260810 23:00:25.063995 17531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380676e85fee4cbebaf30826036f82be" candidate_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013d3b86a63148f4876c2bfe44599056" is_pre_election: true
I20260810 23:00:25.064258 17531 raft_consensus.cc:2454] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4898bdd8ed5d4d5d8b52ab89dd6fe19d in term 0.
I20260810 23:00:25.064271 17668 raft_consensus.cc:2454] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4898bdd8ed5d4d5d8b52ab89dd6fe19d in term 0.
I20260810 23:00:25.064741 17301 leader_election.cc:302] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013d3b86a63148f4876c2bfe44599056, 4898bdd8ed5d4d5d8b52ab89dd6fe19d; no voters:
I20260810 23:00:25.064989 17752 raft_consensus.cc:2781] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:00:25.065073 17752 raft_consensus.cc:482] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:00:25.065116 17752 raft_consensus.cc:3037] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:25.066248 17752 raft_consensus.cc:504] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:25.066565 17752 leader_election.cc:288] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [CANDIDATE]: Term 1 election: Requested vote from peers 37e4ab3f4cfc48388b80de324dbacb4e (127.29.78.131:38469), 013d3b86a63148f4876c2bfe44599056 (127.29.78.130:44043)
I20260810 23:00:25.066892 17668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380676e85fee4cbebaf30826036f82be" candidate_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37e4ab3f4cfc48388b80de324dbacb4e"
I20260810 23:00:25.066892 17531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380676e85fee4cbebaf30826036f82be" candidate_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "013d3b86a63148f4876c2bfe44599056"
I20260810 23:00:25.067016 17531 raft_consensus.cc:3037] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:25.067016 17668 raft_consensus.cc:3037] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:25.068323 17531 raft_consensus.cc:2454] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4898bdd8ed5d4d5d8b52ab89dd6fe19d in term 1.
I20260810 23:00:25.068323 17668 raft_consensus.cc:2454] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4898bdd8ed5d4d5d8b52ab89dd6fe19d in term 1.
I20260810 23:00:25.068620 17301 leader_election.cc:302] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 013d3b86a63148f4876c2bfe44599056, 4898bdd8ed5d4d5d8b52ab89dd6fe19d; no voters:
I20260810 23:00:25.068826 17752 raft_consensus.cc:2781] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:25.069065 17752 raft_consensus.cc:686] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 1 LEADER]: Becoming Leader. State: Replica: 4898bdd8ed5d4d5d8b52ab89dd6fe19d, State: Running, Role: LEADER
I20260810 23:00:25.069375 17752 consensus_queue.cc:237] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } }
I20260810 23:00:25.072540 17217 catalog_manager.cc:5495] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d reported cstate change: term changed from 0 to 1, leader changed from <none> to 4898bdd8ed5d4d5d8b52ab89dd6fe19d (127.29.78.129). New cstate: current_term: 1 leader_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4898bdd8ed5d4d5d8b52ab89dd6fe19d" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 45807 } health_report { overall_health: HEALTHY } } }
I20260810 23:00:25.105260 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:25.101987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52238 (local address 127.29.78.129:45807)
0810 23:00:25.102119 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:25.102123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:25.102137 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:00:25.102267 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:25.102269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:25.102319 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:25.102427 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:00:25.102666 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.103126 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.103917 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:25.104151 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:25.104248 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:25.104534 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:00:25.104879 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:00:25.105001 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:25.105082 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 23:00:25.105127 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:00:25.110656 17668 raft_consensus.cc:1261] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 1 FOLLOWER]: Refusing update from remote peer 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:25.110656 17531 raft_consensus.cc:1261] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 1 FOLLOWER]: Refusing update from remote peer 4898bdd8ed5d4d5d8b52ab89dd6fe19d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:25.111289 17752 consensus_queue.cc:1035] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "37e4ab3f4cfc48388b80de324dbacb4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 38469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:25.111459 17757 consensus_queue.cc:1035] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [LEADER]: Connected to new peer: Peer: permanent_uuid: "013d3b86a63148f4876c2bfe44599056" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:25.117880 17760 mvcc.cc:205] Tried to move back new op lower bound from 7317105971646324736 to 7317105971487051776. Current Snapshot: MvccSnapshot[applied={T|T < 7317105971646324736}]
I20260810 23:00:25.126168 17216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:25.126327 17216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:25.128572 17216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec19ec7c662540c88b753e7c6a95ef56: Sending DeleteTablet for 3 replicas of tablet 380676e85fee4cbebaf30826036f82be
I20260810 23:00:25.129189 17647 tablet_service.cc:1514] Processing DeleteTablet for tablet 380676e85fee4cbebaf30826036f82be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:25 UTC) from {username='slave'} at 127.0.0.1:40550
I20260810 23:00:25.129243 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet 380676e85fee4cbebaf30826036f82be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:25 UTC) from {username='slave'} at 127.0.0.1:33636
I20260810 23:00:25.129580 17355 tablet_service.cc:1514] Processing DeleteTablet for tablet 380676e85fee4cbebaf30826036f82be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:25 UTC) from {username='slave'} at 127.0.0.1:52236
I20260810 23:00:25.129624 17771 tablet_replica.cc:330] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: stopping tablet replica
I20260810 23:00:25.129938 17771 raft_consensus.cc:2229] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:25.129976 17772 tablet_replica.cc:330] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d: stopping tablet replica
I20260810 23:00:25.130213 17772 raft_consensus.cc:2229] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:00:25.130292 17770 tablet_replica.cc:330] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: stopping tablet replica
I20260810 23:00:25.130611 17772 raft_consensus.cc:2258] T 380676e85fee4cbebaf30826036f82be P 4898bdd8ed5d4d5d8b52ab89dd6fe19d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:25.130614 17770 raft_consensus.cc:2229] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:25.130688 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 17282
I20260810 23:00:25.133601 17770 raft_consensus.cc:2258] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:25.134050 17771 raft_consensus.cc:2258] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:25.135159 17770 ts_tablet_manager.cc:1905] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:25.135285 17771 ts_tablet_manager.cc:1905] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:25.137378 17770 ts_tablet_manager.cc:1918] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:25.137490 17770 log.cc:1196] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/380676e85fee4cbebaf30826036f82be
I20260810 23:00:25.137854 17770 ts_tablet_manager.cc:1939] T 380676e85fee4cbebaf30826036f82be P 37e4ab3f4cfc48388b80de324dbacb4e: Deleting consensus metadata
I20260810 23:00:25.138556 17771 ts_tablet_manager.cc:1918] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:25.138651 17771 log.cc:1196] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/380676e85fee4cbebaf30826036f82be
I20260810 23:00:25.138819 17204 catalog_manager.cc:4841] TS 37e4ab3f4cfc48388b80de324dbacb4e (127.29.78.131:38469): tablet 380676e85fee4cbebaf30826036f82be (table test-table [id=3615eebf351c466ca48d9a150f482e7d]) successfully deleted
I20260810 23:00:25.138988 17771 ts_tablet_manager.cc:1939] T 380676e85fee4cbebaf30826036f82be P 013d3b86a63148f4876c2bfe44599056: Deleting consensus metadata
W20260810 23:00:25.139338 17202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:45807 (error 108)
I20260810 23:00:25.139601 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 17432
I20260810 23:00:25.140393 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:25.140126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:45807 (local address 127.0.0.1:52242)
0810 23:00:25.140245 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:25.140313 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 23:00:25.140545 17202 catalog_manager.cc:4584] TS 4898bdd8ed5d4d5d8b52ab89dd6fe19d (127.29.78.129:45807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 380676e85fee4cbebaf30826036f82be: Network error: Client connection negotiation failed: client connection to 127.29.78.129:45807: connect: Connection refused (error 111)
I20260810 23:00:25.146679 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 17586
I20260810 23:00:25.150676 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:25.146894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:44043 (local address 127.0.0.1:33648)
0810 23:00:25.150508 (+ 3614us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:25.150583 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:44043: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3506}
I20260810 23:00:25.155019 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 17186
2026-08-10T23:00:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1329 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:25 dist-test-slave-6nmv krb5kdc[17777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:25 dist-test-slave-6nmv krb5kdc[17777](info): set up 2 sockets
Aug 10 23:00:25 dist-test-slave-6nmv krb5kdc[17777](info): commencing operation
krb5kdc: starting...
W20260810 23:00:26.201871 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:26 dist-test-slave-6nmv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:26.320436 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40837
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:26.424371 17792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:26.424616 17792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:26.424654 17792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:26.428560 17792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:26.428617 17792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:26.428637 17792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:26.428653 17792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:26.428666 17792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:26.433696 17792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:26.434721 17792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:26.435765 17792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:26.443907 17799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:26.443907 17802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:26.444021 17800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:26.444182 17792 server_base.cc:1034] running on GCE node
I20260810 23:00:26.444835 17792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:26.445971 17792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:26.447129 17792 hybrid_clock.cc:648] HybridClock initialized: now 1786402826447113 us; error 31 us; skew 500 ppm
Aug 10 23:00:26 dist-test-slave-6nmv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402826, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:26.450078 17792 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:26.450935 17792 webserver.cc:460] Webserver started at http://127.29.78.190:39053/ using document root <none> and password file <none>
I20260810 23:00:26.451508 17792 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:26.451615 17792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:26.451848 17792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:26.453925 17792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "a904f10910024ae89c357347249fe4b7"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "23f328dbcebcc9c157390ce2435c3927"
server_key_iv: "df4602a68cc2ccc2392ff58974f6c1e4"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.454506 17792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "a904f10910024ae89c357347249fe4b7"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "23f328dbcebcc9c157390ce2435c3927"
server_key_iv: "df4602a68cc2ccc2392ff58974f6c1e4"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.458181 17792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:00:26.460500 17810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:26.461663 17792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:26.461844 17792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "a904f10910024ae89c357347249fe4b7"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "23f328dbcebcc9c157390ce2435c3927"
server_key_iv: "df4602a68cc2ccc2392ff58974f6c1e4"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.461964 17792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:26.478261 17792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:26.481423 17792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:26.481627 17792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:26.488621 17792 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40837
I20260810 23:00:26.488633 17866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40837 every 8 connection(s)
I20260810 23:00:26.489548 17792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:26.492303 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:26.496230 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 17792
I20260810 23:00:26.496387 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:26.496685 30010 external_mini_cluster.cc:1428] Setting key 09d902f1e496e3eb7d1326c86976130d
I20260810 23:00:26.497740 17867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7: Bootstrap starting.
I20260810 23:00:26.500415 17867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:26.501386 17867 log.cc:824] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:26.503322 17867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7: No bootstrap required, opened a new log
Aug 10 23:00:26 dist-test-slave-6nmv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:26.507706 17867 raft_consensus.cc:348] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a904f10910024ae89c357347249fe4b7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40837 } }
I20260810 23:00:26.507968 17867 raft_consensus.cc:374] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:26.508041 17867 raft_consensus.cc:729] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a904f10910024ae89c357347249fe4b7, State: Initialized, Role: FOLLOWER
I20260810 23:00:26.508548 17867 consensus_queue.cc:260] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a904f10910024ae89c357347249fe4b7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40837 } }
I20260810 23:00:26.508697 17867 raft_consensus.cc:388] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:26.508795 17867 raft_consensus.cc:482] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:26.508898 17867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:26.509981 17867 raft_consensus.cc:504] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a904f10910024ae89c357347249fe4b7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40837 } }
I20260810 23:00:26.510303 17867 leader_election.cc:302] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a904f10910024ae89c357347249fe4b7; no voters:
I20260810 23:00:26.510628 17867 leader_election.cc:288] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:26.510771 17872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:26.511036 17872 raft_consensus.cc:686] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [term 1 LEADER]: Becoming Leader. State: Replica: a904f10910024ae89c357347249fe4b7, State: Running, Role: LEADER
I20260810 23:00:26.511386 17872 consensus_queue.cc:237] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a904f10910024ae89c357347249fe4b7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40837 } }
I20260810 23:00:26.511409 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:26.498073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46376 (local address 127.29.78.190:40837)
0810 23:00:26.498461 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:26.498479 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:26.498520 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:00:26.499348 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:26.499368 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:26.499750 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:26.500096 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:26.500866 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.502233 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.503020 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.503626 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.503786 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:26.506370 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:26.506378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:26.506395 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:26.506426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:26.508358 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:26.508790 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:26.508795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:26.508801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:26.508966 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:26.509227 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:26.509230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:26.509231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:26.509696 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:26.509918 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:26.510136 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 23:00:26.510403 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":111,"threads_started":1}
I20260810 23:00:26.511868 17867 sys_catalog.cc:564] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:26.514061 17873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a904f10910024ae89c357347249fe4b7. Latest consensus state: current_term: 1 leader_uuid: "a904f10910024ae89c357347249fe4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a904f10910024ae89c357347249fe4b7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40837 } } }
I20260810 23:00:26.514238 17875 sys_catalog.cc:455] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a904f10910024ae89c357347249fe4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a904f10910024ae89c357347249fe4b7" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40837 } } }
I20260810 23:00:26.514348 17873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:26.514358 17875 sys_catalog.cc:458] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7 [sys.catalog]: This master's current role is: LEADER
W20260810 23:00:26.516768 17888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:00:26.516849 17888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:00:26.516906 17889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:26.517786 17889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:26.522692 17889 catalog_manager.cc:1347] Generated new cluster ID: c5a91dfc17de4e2ba8c6ab0fe671017d
I20260810 23:00:26.522753 17889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:26.532063 17889 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:26.533020 17889 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:26.538960 17889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7: Generated new TSK 0
I20260810 23:00:26.539611 17889 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 23:00:26.573074 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40837
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:26.696085 17895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:26.696331 17895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:26.696370 17895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:26.700366 17895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:26.700418 17895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:26.700532 17895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 23:00:26.705415 17895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:26.706383 17895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:26.707319 17895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:26.716423 17903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:26.716389 17900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:26.716442 17901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:26.716642 17895 server_base.cc:1034] running on GCE node
I20260810 23:00:26.717039 17895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:26.717617 17895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:26.718793 17895 hybrid_clock.cc:648] HybridClock initialized: now 1786402826718761 us; error 46 us; skew 500 ppm
Aug 10 23:00:26 dist-test-slave-6nmv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402826, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:26.721223 17895 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 23:00:26.722111 17895 webserver.cc:460] Webserver started at http://127.29.78.129:38549/ using document root <none> and password file <none>
I20260810 23:00:26.722606 17895 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:26.722693 17895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:26.722903 17895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:26.724676 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "071a74740ded4d41ac628ceface83b9f"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "795cd30902596406a039d7d2536cef0c"
server_key_iv: "59091af25ff7d8b121a69dfe7df179d9"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.725200 17895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "071a74740ded4d41ac628ceface83b9f"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "795cd30902596406a039d7d2536cef0c"
server_key_iv: "59091af25ff7d8b121a69dfe7df179d9"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.728575 17895 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:00:26.730842 17911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:26.731976 17895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:26.732105 17895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "071a74740ded4d41ac628ceface83b9f"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "795cd30902596406a039d7d2536cef0c"
server_key_iv: "59091af25ff7d8b121a69dfe7df179d9"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.732208 17895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:26.752712 17895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:26.755312 17895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:26.755514 17895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:26.756027 17895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:26.756809 17895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:26.756882 17895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:26.756953 17895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:26.757002 17895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:26.765057 17895 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:43061
I20260810 23:00:26.765108 18033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:43061 every 8 connection(s)
I20260810 23:00:26.765844 17895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 23:00:26.770264 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 17895
I20260810 23:00:26.770409 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 23:00:26.770668 30010 external_mini_cluster.cc:1428] Setting key 5376f92328734e2c8a13fdf87946c526
Aug 10 23:00:26 dist-test-slave-6nmv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402826, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:26.778818 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:26.767575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:42809 (local address 127.29.78.190:40837)
0810 23:00:26.767721 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:26.767725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:26.768441 (+ 716us) server_negotiation.cc:407] Connection header received
0810 23:00:26.769568 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:26.769570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:26.769614 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:26.769684 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:26.770845 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.771435 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.772613 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.772857 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.772958 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:26.775344 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:26.775347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:26.775349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:26.775369 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:26.776723 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:26.777228 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:26.777232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:26.777234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:26.777351 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:26.777708 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:26.777712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:26.777713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:26.777945 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:26.778055 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:26.778522 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 23:00:26.778661 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:00:26.779536 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:26.767911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40837 (local address 127.29.78.129:42809)
0810 23:00:26.768313 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:26.768350 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:26.769394 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:26.769873 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:26.769879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:26.770198 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:26.770682 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:26.770695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.771598 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.771602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:26.772481 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:26.772490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.773011 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.773015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:26.773185 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:26.773718 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:26.773733 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:26.775183 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:26.776844 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:26.776850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:26.776862 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:26.777120 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:26.777476 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:26.777478 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:26.777480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:26.777612 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:26.778052 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:26.778057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:26.778281 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 23:00:26.778484 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 23:00:26.778782 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 23:00:26.780761 18034 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40837
I20260810 23:00:26.780997 18034 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:26.781423 18034 heartbeater.cc:511] Master 127.29.78.190:40837 requested a full tablet report, sending...
I20260810 23:00:26.782876 17829 ts_manager.cc:194] Registered new tserver with Master: 071a74740ded4d41ac628ceface83b9f (127.29.78.129:43061)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
I20260810 23:00:26.784204 17829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:42809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 23:00:26.798383 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40837
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:26.898269 18039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:26.898494 18039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:26.898531 18039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:26.902179 18039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:26.902223 18039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:26.902275 18039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 23:00:26.907130 18039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:26.908028 18039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:26.908900 18039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:26.918092 18047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:26.918198 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:26.918090 18044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:26.918403 18039 server_base.cc:1034] running on GCE node
I20260810 23:00:26.918741 18039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:26.919301 18039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:26.920482 18039 hybrid_clock.cc:648] HybridClock initialized: now 1786402826920462 us; error 55 us; skew 500 ppm
Aug 10 23:00:26 dist-test-slave-6nmv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402826, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:26.923272 18039 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 23:00:26.924058 18039 webserver.cc:460] Webserver started at http://127.29.78.130:45449/ using document root <none> and password file <none>
I20260810 23:00:26.924547 18039 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:26.924628 18039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:26.924825 18039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:26.926748 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "c14b8e66292548d7be469117df1160ea"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "578eb30220a59b59a103e0bacd8c5ee2"
server_key_iv: "d025f029b1060ab03869528f0bef0624"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.927297 18039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "c14b8e66292548d7be469117df1160ea"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "578eb30220a59b59a103e0bacd8c5ee2"
server_key_iv: "d025f029b1060ab03869528f0bef0624"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.930601 18039 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:00:26.932926 18054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:26.934051 18039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:26.934171 18039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "c14b8e66292548d7be469117df1160ea"
format_stamp: "Formatted at 2026-08-10 23:00:26 on dist-test-slave-6nmv"
server_key: "578eb30220a59b59a103e0bacd8c5ee2"
server_key_iv: "d025f029b1060ab03869528f0bef0624"
server_key_version: "encryptionkey@0"
I20260810 23:00:26.934269 18039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:26.945670 18039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:26.948673 18039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:26.948848 18039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:26.949388 18039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:26.950203 18039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:26.950273 18039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:26.950340 18039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:26.950387 18039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:26.958662 18039 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:34255
I20260810 23:00:26.958694 18181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:34255 every 8 connection(s)
I20260810 23:00:26.959404 18039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 23:00:26.963928 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 18039
I20260810 23:00:26.964067 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 23:00:26.964326 30010 external_mini_cluster.cc:1428] Setting key 7da499280a8fb1738b29ca90e7a674c8
Aug 10 23:00:26 dist-test-slave-6nmv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402826, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:26.971889 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:26.961011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:41543 (local address 127.29.78.190:40837)
0810 23:00:26.961135 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:26.961139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:26.961963 (+ 824us) server_negotiation.cc:407] Connection header received
0810 23:00:26.963163 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:26.963166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:26.963218 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:26.963310 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:26.964403 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.964885 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.966146 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.966375 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.966510 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:26.968728 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:26.968733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:26.968740 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:26.968760 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:26.970044 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:26.970510 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:26.970514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:26.970516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:26.970591 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:26.970898 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:26.970902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:26.970903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:26.971065 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:26.971168 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:26.971603 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 23:00:26.971732 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:00:26.972572 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:26.961382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40837 (local address 127.29.78.130:41543)
0810 23:00:26.961783 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:26.961862 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:26.962971 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:26.963489 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:26.963495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:26.963783 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:26.964263 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:26.964276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.965007 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.965010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:26.966018 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:26.966026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:26.966524 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:26.966528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:26.966701 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:26.967252 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:26.967267 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:26.968582 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:26.970144 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:26.970149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:26.970160 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:26.970406 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:26.970676 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:26.970679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:26.970681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:26.970806 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:26.971166 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:26.971172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:26.971377 (+ 205us) client_negotiation.cc:769] Sending connection context
0810 23:00:26.971550 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 23:00:26.971855 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":103,"threads_started":1}
I20260810 23:00:26.973637 18182 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40837
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 23:00:26.973865 18182 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:26.974290 18182 heartbeater.cc:511] Master 127.29.78.190:40837 requested a full tablet report, sending...
I20260810 23:00:26.975368 17829 ts_manager.cc:194] Registered new tserver with Master: c14b8e66292548d7be469117df1160ea (127.29.78.130:34255)
I20260810 23:00:26.975956 17829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:41543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 23:00:26.990345 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40837
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:27.094532 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:27.094811 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:27.094851 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:27.098711 18187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:27.098762 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:27.098846 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 23:00:27.103569 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:27.104432 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:27.105346 18187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:27.114279 18192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:27.114282 18193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:27.114506 18195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:27.114622 18187 server_base.cc:1034] running on GCE node
I20260810 23:00:27.114995 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:27.115500 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:27.116654 18187 hybrid_clock.cc:648] HybridClock initialized: now 1786402827116638 us; error 25 us; skew 500 ppm
Aug 10 23:00:27 dist-test-slave-6nmv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402827, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:27.119439 18187 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 23:00:27.120231 18187 webserver.cc:460] Webserver started at http://127.29.78.131:36199/ using document root <none> and password file <none>
I20260810 23:00:27.120723 18187 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:27.120806 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:27.121035 18187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:27.122841 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "fe4c93d0fc194e9183512afbb40fdc4e"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "3852e5fe6146f47f1bf55f7a5ff3a6e7"
server_key_iv: "b581794beab9e437cebe6f3487016894"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.123337 18187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "fe4c93d0fc194e9183512afbb40fdc4e"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "3852e5fe6146f47f1bf55f7a5ff3a6e7"
server_key_iv: "b581794beab9e437cebe6f3487016894"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.126823 18187 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:00:27.129088 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.130244 18187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:27.130364 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "fe4c93d0fc194e9183512afbb40fdc4e"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "3852e5fe6146f47f1bf55f7a5ff3a6e7"
server_key_iv: "b581794beab9e437cebe6f3487016894"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.130460 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:27.149166 18187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:27.152297 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:27.152498 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:27.153029 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:27.153785 18187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:27.153889 18187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.153975 18187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:27.154021 18187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.162642 18187 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:41571
I20260810 23:00:27.162669 18331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:41571 every 8 connection(s)
I20260810 23:00:27.163443 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 23:00:27.165660 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 18187
I20260810 23:00:27.165791 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 23:00:27.166115 30010 external_mini_cluster.cc:1428] Setting key 1278cfd44b6cde5531df755075d98ccd
Aug 10 23:00:27 dist-test-slave-6nmv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402827, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:27.176087 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.165101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:36271 (local address 127.29.78.190:40837)
0810 23:00:27.165234 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.165239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.166124 (+ 885us) server_negotiation.cc:407] Connection header received
0810 23:00:27.167747 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.167750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.167798 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.167882 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:27.168995 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.169493 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.170500 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.170702 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.170778 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.173026 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:27.173029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:27.173030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:27.173050 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:27.174439 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:27.174838 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:27.174840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:27.174841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:27.174908 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:27.175237 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:27.175238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:27.175239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:27.175382 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:27.175449 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.175870 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.175978 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:00:27.176949 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.165510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40837 (local address 127.29.78.131:36271)
0810 23:00:27.165992 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.166039 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:27.167578 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:27.167992 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:27.167998 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:27.168433 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:27.168868 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.168881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.169589 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.169591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.170376 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.170384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.170821 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.170823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.170984 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.171466 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:27.171480 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:27.172881 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:27.174531 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:27.174535 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:27.174544 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:27.174765 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:27.175013 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:27.175015 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:27.175016 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:27.175153 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:27.175466 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:27.175470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:27.175661 (+ 191us) client_negotiation.cc:769] Sending connection context
0810 23:00:27.175858 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 23:00:27.176106 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
I20260810 23:00:27.178107 18332 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40837
I20260810 23:00:27.178326 18332 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:27.178788 18332 heartbeater.cc:511] Master 127.29.78.190:40837 requested a full tablet report, sending...
I20260810 23:00:27.179881 17828 ts_manager.cc:194] Registered new tserver with Master: fe4c93d0fc194e9183512afbb40fdc4e (127.29.78.131:41571)
I20260810 23:00:27.180419 17828 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:36271
I20260810 23:00:27.191238 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:00:27.200289 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.193098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.29.78.190:40837)
0810 23:00:27.193212 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.193215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.193387 (+ 172us) server_negotiation.cc:407] Connection header received
0810 23:00:27.193483 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.193484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.193517 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.193581 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:27.194232 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.194676 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.195376 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.195565 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.195647 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.197227 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:27.197230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:27.197232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:27.197254 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:27.198728 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:27.199205 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:27.199208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:27.199209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:27.199279 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:27.199555 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:27.199557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:27.199558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:27.199769 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:27.199879 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.200092 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.200164 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:00:27.203162 17828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:00:27.206084 17828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:00:27.220395 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.214115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:34255 (local address 127.0.0.1:56464)
0810 23:00:27.214715 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.214763 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:27.214894 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:27.215935 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:27.215942 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:27.215973 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:27.216116 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.216124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.217670 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.217673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.218976 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.218985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.219973 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.219976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.220061 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.220088 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:00:27.220165 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:00:27.220237 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":65,"thread_start_us":84,"threads_started":1}
I20260810 23:00:27.220440 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.214207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.29.78.130:34255)
0810 23:00:27.215558 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.215568 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.215588 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:00:27.215646 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.215649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.215792 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.215902 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:27.216303 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.217514 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.219101 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.219849 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.219969 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.220119 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.220210 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.220313 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1245,"thread_start_us":125,"threads_started":1}
I20260810 23:00:27.222016 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.214196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:41571 (local address 127.0.0.1:44358)
0810 23:00:27.215186 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.215201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:27.215297 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:27.216070 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:27.216073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:27.216090 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:27.216172 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.216178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.217670 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.217673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.219288 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.219294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.221388 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.221392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.221484 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.221505 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:00:27.221737 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 23:00:27.221838 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":230,"thread_start_us":83,"threads_started":1}
I20260810 23:00:27.222038 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.214913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.29.78.131:41571)
0810 23:00:27.215680 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.215687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.215703 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:00:27.215757 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.215760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.215932 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.216068 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:27.216315 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.217514 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.220530 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.221271 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.221385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.221529 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.221771 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.221910 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":169,"threads_started":1}
I20260810 23:00:27.223426 18107 tablet_service.cc:1467] Processing CreateTablet for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 (DEFAULT_TABLE table=test-table [id=531288fc027c40c5a625bc6ff70159c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:27.224308 18107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:27.224184 18253 tablet_service.cc:1467] Processing CreateTablet for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 (DEFAULT_TABLE table=test-table [id=531288fc027c40c5a625bc6ff70159c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:27.225028 18253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:27.226123 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.214452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43061 (local address 127.0.0.1:45520)
0810 23:00:27.215374 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.215387 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:27.215478 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:27.216566 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:27.216569 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:27.216585 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:27.216668 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.216673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.217676 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.217679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.219271 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.219282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.220352 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.220355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.220440 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.220469 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 23:00:27.225933 (+ 5464us) client_negotiation.cc:240] Negotiation successful
0810 23:00:27.226008 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"mutex_wait_us":182,"thread_start_us":82,"threads_started":1}
I20260810 23:00:27.226629 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.214906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.29.78.129:43061)
0810 23:00:27.216178 (+ 1272us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.216207 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.216229 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:00:27.216304 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.216309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.216442 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.216587 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:27.216784 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.217524 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.219402 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.220239 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.220335 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.220455 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.226373 (+ 5918us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.226492 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169,"thread_start_us":109,"threads_started":1}
I20260810 23:00:27.229314 17964 tablet_service.cc:1467] Processing CreateTablet for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 (DEFAULT_TABLE table=test-table [id=531288fc027c40c5a625bc6ff70159c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:27.230475 17964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:27.231316 18351 tablet_bootstrap.cc:492] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Bootstrap starting.
I20260810 23:00:27.232724 18353 tablet_bootstrap.cc:492] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Bootstrap starting.
I20260810 23:00:27.233698 18351 tablet_bootstrap.cc:654] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:27.234988 18353 tablet_bootstrap.cc:654] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:27.235714 18351 log.cc:824] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:27.236384 18353 log.cc:824] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:27.237514 18356 tablet_bootstrap.cc:492] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: Bootstrap starting.
I20260810 23:00:27.238370 18351 tablet_bootstrap.cc:492] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: No bootstrap required, opened a new log
I20260810 23:00:27.238464 18353 tablet_bootstrap.cc:492] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: No bootstrap required, opened a new log
I20260810 23:00:27.238577 18351 ts_tablet_manager.cc:1397] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 23:00:27.238653 18353 ts_tablet_manager.cc:1397] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260810 23:00:27.239542 18356 tablet_bootstrap.cc:654] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:27.240549 18356 log.cc:824] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:27.242448 18356 tablet_bootstrap.cc:492] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: No bootstrap required, opened a new log
I20260810 23:00:27.242465 18351 raft_consensus.cc:348] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.242465 18353 raft_consensus.cc:348] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.242667 18356 ts_tablet_manager.cc:1397] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260810 23:00:27.242730 18353 raft_consensus.cc:374] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:27.242808 18353 raft_consensus.cc:729] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4c93d0fc194e9183512afbb40fdc4e, State: Initialized, Role: FOLLOWER
I20260810 23:00:27.242887 18351 raft_consensus.cc:374] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:27.242990 18351 raft_consensus.cc:729] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c14b8e66292548d7be469117df1160ea, State: Initialized, Role: FOLLOWER
I20260810 23:00:27.243384 18353 consensus_queue.cc:260] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.243407 18351 consensus_queue.cc:260] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.244046 18182 heartbeater.cc:503] Master 127.29.78.190:40837 was elected leader, sending a full tablet report...
I20260810 23:00:27.244300 18351 ts_tablet_manager.cc:1428] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 23:00:27.245275 18353 ts_tablet_manager.cc:1428] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20260810 23:00:27.245635 18332 heartbeater.cc:503] Master 127.29.78.190:40837 was elected leader, sending a full tablet report...
I20260810 23:00:27.246038 18356 raft_consensus.cc:348] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.246285 18356 raft_consensus.cc:374] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:27.246342 18356 raft_consensus.cc:729] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 071a74740ded4d41ac628ceface83b9f, State: Initialized, Role: FOLLOWER
I20260810 23:00:27.246809 18356 consensus_queue.cc:260] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.247666 18034 heartbeater.cc:503] Master 127.29.78.190:40837 was elected leader, sending a full tablet report...
I20260810 23:00:27.247851 18356 ts_tablet_manager.cc:1428] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
W20260810 23:00:27.267006 18035 tablet.cc:2369] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:27.312789 18358 raft_consensus.cc:482] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:27.313011 18358 raft_consensus.cc:504] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.314232 18358 leader_election.cc:288] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe4c93d0fc194e9183512afbb40fdc4e (127.29.78.131:41571), 071a74740ded4d41ac628ceface83b9f (127.29.78.129:43061)
I20260810 23:00:27.318096 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.314573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:60501 (local address 127.29.78.129:43061)
0810 23:00:27.314693 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.314698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.314713 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:00:27.314791 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.314794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.314844 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.314930 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:27.315274 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.315871 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.317029 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.317487 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.317577 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.317655 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.317907 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.317971 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:00:27.318113 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.314584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49805 (local address 127.29.78.131:41571)
0810 23:00:27.314694 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.314698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.315016 (+ 318us) server_negotiation.cc:407] Connection header received
0810 23:00:27.315140 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.315143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.315196 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.315272 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:27.315531 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.316051 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.317135 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.317553 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.317657 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.317709 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.317934 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.317998 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:00:27.318114 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.314458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43061 (local address 127.29.78.130:60501)
0810 23:00:27.314561 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.314577 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:27.314666 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:27.314960 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:27.314963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:27.314987 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:27.315129 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.315135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.315983 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.315987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.316888 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.316897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.317609 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.317615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.317744 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.317765 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:00:27.317887 (+ 122us) client_negotiation.cc:240] Negotiation successful
0810 23:00:27.317959 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":47,"spinlock_wait_cycles":21376}
I20260810 23:00:27.318145 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.314458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:41571 (local address 127.29.78.130:49805)
0810 23:00:27.314910 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.314925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:27.315053 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:27.315294 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:27.315297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:27.315314 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:27.315426 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.315432 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.316162 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.316165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.317013 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.317024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.317688 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.317691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.317776 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.317793 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 23:00:27.317901 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:00:27.317984 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":120,"spinlock_wait_cycles":3200,"thread_start_us":122,"threads_started":1}
I20260810 23:00:27.318687 18279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca06ea8d1a8a4f1e8063b8b27dc8e308" candidate_uuid: "c14b8e66292548d7be469117df1160ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" is_pre_election: true
I20260810 23:00:27.318995 18279 raft_consensus.cc:2454] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c14b8e66292548d7be469117df1160ea in term 0.
I20260810 23:00:27.319056 17985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca06ea8d1a8a4f1e8063b8b27dc8e308" candidate_uuid: "c14b8e66292548d7be469117df1160ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071a74740ded4d41ac628ceface83b9f" is_pre_election: true
I20260810 23:00:27.319319 17985 raft_consensus.cc:2454] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c14b8e66292548d7be469117df1160ea in term 0.
I20260810 23:00:27.319783 18059 leader_election.cc:302] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 071a74740ded4d41ac628ceface83b9f, c14b8e66292548d7be469117df1160ea, fe4c93d0fc194e9183512afbb40fdc4e; no voters:
I20260810 23:00:27.320056 18358 raft_consensus.cc:2781] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:00:27.320154 18358 raft_consensus.cc:482] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:00:27.320211 18358 raft_consensus.cc:3037] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:27.321208 18358 raft_consensus.cc:504] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.321537 18358 leader_election.cc:288] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [CANDIDATE]: Term 1 election: Requested vote from peers fe4c93d0fc194e9183512afbb40fdc4e (127.29.78.131:41571), 071a74740ded4d41ac628ceface83b9f (127.29.78.129:43061)
I20260810 23:00:27.321981 17985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca06ea8d1a8a4f1e8063b8b27dc8e308" candidate_uuid: "c14b8e66292548d7be469117df1160ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "071a74740ded4d41ac628ceface83b9f"
I20260810 23:00:27.322101 17985 raft_consensus.cc:3037] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:27.323045 17985 raft_consensus.cc:2454] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c14b8e66292548d7be469117df1160ea in term 1.
I20260810 23:00:27.323335 18279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca06ea8d1a8a4f1e8063b8b27dc8e308" candidate_uuid: "c14b8e66292548d7be469117df1160ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe4c93d0fc194e9183512afbb40fdc4e"
I20260810 23:00:27.323369 18059 leader_election.cc:302] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 071a74740ded4d41ac628ceface83b9f, c14b8e66292548d7be469117df1160ea; no voters:
I20260810 23:00:27.323462 18279 raft_consensus.cc:3037] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:27.324398 18279 raft_consensus.cc:2454] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c14b8e66292548d7be469117df1160ea in term 1.
I20260810 23:00:27.324587 18358 raft_consensus.cc:2781] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:27.324829 18358 raft_consensus.cc:686] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 1 LEADER]: Becoming Leader. State: Replica: c14b8e66292548d7be469117df1160ea, State: Running, Role: LEADER
I20260810 23:00:27.325179 18358 consensus_queue.cc:237] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } }
I20260810 23:00:27.328190 17829 catalog_manager.cc:5495] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea reported cstate change: term changed from 0 to 1, leader changed from <none> to c14b8e66292548d7be469117df1160ea (127.29.78.130). New cstate: current_term: 1 leader_uuid: "c14b8e66292548d7be469117df1160ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c14b8e66292548d7be469117df1160ea" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 } health_report { overall_health: UNKNOWN } } }
I20260810 23:00:27.341940 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.337700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.29.78.130:34255)
0810 23:00:27.337853 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.337858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.337877 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:00:27.338046 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.338049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.338105 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.338204 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:00:27.338418 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.339146 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.340494 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.340690 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.340784 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.341025 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:00:27.341547 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:00:27.341679 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.341746 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.341779 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:00:27.347455 18279 raft_consensus.cc:1261] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 1 FOLLOWER]: Refusing update from remote peer c14b8e66292548d7be469117df1160ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:27.347450 17985 raft_consensus.cc:1261] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 1 FOLLOWER]: Refusing update from remote peer c14b8e66292548d7be469117df1160ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:27.348239 18363 consensus_queue.cc:1035] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "071a74740ded4d41ac628ceface83b9f" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 43061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:27.348483 18358 consensus_queue.cc:1035] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe4c93d0fc194e9183512afbb40fdc4e" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 41571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:27.365561 17828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:27.365720 17828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:27.368034 17828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a904f10910024ae89c357347249fe4b7: Sending DeleteTablet for 3 replicas of tablet ca06ea8d1a8a4f1e8063b8b27dc8e308
I20260810 23:00:27.368768 18107 tablet_service.cc:1514] Processing DeleteTablet for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:27 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:56464
I20260810 23:00:27.368769 18253 tablet_service.cc:1514] Processing DeleteTablet for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:27 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:44358
I20260810 23:00:27.369269 18376 tablet_replica.cc:330] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: stopping tablet replica
I20260810 23:00:27.369262 17964 tablet_service.cc:1514] Processing DeleteTablet for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:27 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:45520
I20260810 23:00:27.369577 18376 raft_consensus.cc:2229] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:00:27.369710 18378 tablet_replica.cc:330] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f: stopping tablet replica
I20260810 23:00:27.370057 18376 raft_consensus.cc:2258] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:27.370069 18378 raft_consensus.cc:2229] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:27.370375 18378 raft_consensus.cc:2258] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P 071a74740ded4d41ac628ceface83b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:27.370471 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 17895
I20260810 23:00:27.376420 18376 ts_tablet_manager.cc:1905] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:27.376829 18377 tablet_replica.cc:330] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: stopping tablet replica
I20260810 23:00:27.377220 18377 raft_consensus.cc:2229] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:27.377633 18377 raft_consensus.cc:2258] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:27.379197 18377 ts_tablet_manager.cc:1905] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:27.379310 18376 ts_tablet_manager.cc:1918] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:27.379465 18376 log.cc:1196] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/ca06ea8d1a8a4f1e8063b8b27dc8e308
I20260810 23:00:27.379850 18376 ts_tablet_manager.cc:1939] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P c14b8e66292548d7be469117df1160ea: Deleting consensus metadata
I20260810 23:00:27.381062 17813 catalog_manager.cc:4841] TS c14b8e66292548d7be469117df1160ea (127.29.78.130:34255): tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 (table test-table [id=531288fc027c40c5a625bc6ff70159c5]) successfully deleted
W20260810 23:00:27.381438 17814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:43061 (error 108)
I20260810 23:00:27.382270 18377 ts_tablet_manager.cc:1918] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:27.382381 18377 log.cc:1196] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/ca06ea8d1a8a4f1e8063b8b27dc8e308
I20260810 23:00:27.382491 18345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.382173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:43061 (local address 127.0.0.1:45536)
0810 23:00:27.382303 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.382395 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:43061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:00:27.382669 17814 catalog_manager.cc:4584] TS 071a74740ded4d41ac628ceface83b9f (127.29.78.129:43061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca06ea8d1a8a4f1e8063b8b27dc8e308: Network error: Client connection negotiation failed: client connection to 127.29.78.129:43061: connect: Connection refused (error 111)
I20260810 23:00:27.382733 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 18039
I20260810 23:00:27.382766 18377 ts_tablet_manager.cc:1939] T ca06ea8d1a8a4f1e8063b8b27dc8e308 P fe4c93d0fc194e9183512afbb40fdc4e: Deleting consensus metadata
I20260810 23:00:27.384080 17811 catalog_manager.cc:4841] TS fe4c93d0fc194e9183512afbb40fdc4e (127.29.78.131:41571): tablet ca06ea8d1a8a4f1e8063b8b27dc8e308 (table test-table [id=531288fc027c40c5a625bc6ff70159c5]) successfully deleted
I20260810 23:00:27.389729 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 18187
I20260810 23:00:27.396073 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 17792
2026-08-10T23:00:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2243 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T23:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:27Z Disabled control of system clock
I20260810 23:00:27.428990 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36039
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36039
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:00:27.529475 18383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:27.529744 18383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:27.529855 18383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:27.533797 18383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:27.533890 18383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:27.533917 18383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:27.533962 18383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:27.533990 18383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:27.538545 18383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36039
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:27.539630 18383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:27.540971 18383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:27.549439 18390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:27.549432 18389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:27.549481 18383 server_base.cc:1034] running on GCE node
W20260810 23:00:27.549432 18393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:27.550355 18383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:27.551417 18383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:27.552582 18383 hybrid_clock.cc:648] HybridClock initialized: now 1786402827552562 us; error 28 us; skew 500 ppm
I20260810 23:00:27.554507 18383 webserver.cc:460] Webserver started at http://127.29.78.190:44309/ using document root <none> and password file <none>
I20260810 23:00:27.555011 18383 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:27.555075 18383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:27.555233 18383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:27.557176 18383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "8bef78c118004eadb8bc994060c7701e"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "e97e34ffe3c6011f69707b7d899d1d56"
server_key_iv: "5f182319609d26a1e536bbc73ae950c1"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.557680 18383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "8bef78c118004eadb8bc994060c7701e"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "e97e34ffe3c6011f69707b7d899d1d56"
server_key_iv: "5f182319609d26a1e536bbc73ae950c1"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.561174 18383 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:00:27.563454 18400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.564630 18383 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:27.564745 18383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "8bef78c118004eadb8bc994060c7701e"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "e97e34ffe3c6011f69707b7d899d1d56"
server_key_iv: "5f182319609d26a1e536bbc73ae950c1"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.564869 18383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:27.583632 18383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:27.584275 18383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:27.584476 18383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:27.590873 18383 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36039
I20260810 23:00:27.590941 18458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36039 every 8 connection(s)
I20260810 23:00:27.591797 18383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:27.594281 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 18383
I20260810 23:00:27.594394 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:27.594698 30010 external_mini_cluster.cc:1428] Setting key c3541ed5c9ec2b35435a5157a3b7377c
I20260810 23:00:27.594883 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:27.600950 18459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e: Bootstrap starting.
I20260810 23:00:27.603277 18459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:27.603478 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.596078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.29.78.190:36039)
0810 23:00:27.596763 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.596777 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.596811 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:00:27.597434 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.597450 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.597891 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.598207 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:27.598606 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.599525 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.600468 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.600886 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.601054 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.601889 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:27.601893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:27.601903 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:27.602234 (+ 331us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:27.602283 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:27.602300 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:27.602422 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.602558 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.602784 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":172,"threads_started":1}
I20260810 23:00:27.604074 18459 log.cc:824] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:27.605985 18459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e: No bootstrap required, opened a new log
I20260810 23:00:27.609092 18459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef78c118004eadb8bc994060c7701e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36039 } }
I20260810 23:00:27.609297 18459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:27.609366 18459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bef78c118004eadb8bc994060c7701e, State: Initialized, Role: FOLLOWER
I20260810 23:00:27.609862 18459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef78c118004eadb8bc994060c7701e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36039 } }
I20260810 23:00:27.610016 18459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:27.610085 18459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:27.610172 18459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:27.611061 18459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef78c118004eadb8bc994060c7701e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36039 } }
I20260810 23:00:27.611393 18459 leader_election.cc:302] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bef78c118004eadb8bc994060c7701e; no voters:
I20260810 23:00:27.611743 18459 leader_election.cc:288] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:27.611825 18464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:27.612083 18464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [term 1 LEADER]: Becoming Leader. State: Replica: 8bef78c118004eadb8bc994060c7701e, State: Running, Role: LEADER
I20260810 23:00:27.612375 18464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef78c118004eadb8bc994060c7701e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36039 } }
I20260810 23:00:27.612978 18459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:27.613942 18467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bef78c118004eadb8bc994060c7701e. Latest consensus state: current_term: 1 leader_uuid: "8bef78c118004eadb8bc994060c7701e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef78c118004eadb8bc994060c7701e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36039 } } }
I20260810 23:00:27.614075 18467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:27.614315 18466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bef78c118004eadb8bc994060c7701e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bef78c118004eadb8bc994060c7701e" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36039 } } }
I20260810 23:00:27.614396 18466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:27.614383 18476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:27.617188 18476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:27.622655 18476 catalog_manager.cc:1347] Generated new cluster ID: f6e61a2f1b784e93a0afa9ac5f7db7ce
I20260810 23:00:27.622726 18476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:27.636390 18476 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:27.637274 18476 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:27.649274 18476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e: Generated new TSK 0
I20260810 23:00:27.649920 18476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:27.657842 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36039
--builtin_ntp_servers=127.29.78.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:00:27.764832 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:27.765043 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:27.765076 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:27.769095 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:27.769143 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:27.769232 18489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 23:00:27.773937 18489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:27.774821 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:27.776010 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:27.785147 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:27.785180 18494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:27.785351 18498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:27.785497 18489 server_base.cc:1034] running on GCE node
I20260810 23:00:27.785943 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:27.786561 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:27.787752 18489 hybrid_clock.cc:648] HybridClock initialized: now 1786402827787734 us; error 29 us; skew 500 ppm
I20260810 23:00:27.789536 18489 webserver.cc:460] Webserver started at http://127.29.78.129:45023/ using document root <none> and password file <none>
I20260810 23:00:27.790220 18489 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:27.790316 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:27.790547 18489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:27.792538 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "1b06ce702bc141f99bd10f9c1960a0c4"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "e47e8c47b1d41e640be09b653cea8b78"
server_key_iv: "0862fc41b80cc66a00633f95e7f1cfc9"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.793107 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "1b06ce702bc141f99bd10f9c1960a0c4"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "e47e8c47b1d41e640be09b653cea8b78"
server_key_iv: "0862fc41b80cc66a00633f95e7f1cfc9"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.796541 18489 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:00:27.798755 18507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.799794 18489 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:27.799933 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "1b06ce702bc141f99bd10f9c1960a0c4"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "e47e8c47b1d41e640be09b653cea8b78"
server_key_iv: "0862fc41b80cc66a00633f95e7f1cfc9"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.800050 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:27.813253 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:27.813936 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:27.814078 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:27.814611 18489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:27.815359 18489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:27.815403 18489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.815436 18489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:27.815459 18489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.823298 18489 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:36431
I20260810 23:00:27.823318 18633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:36431 every 8 connection(s)
I20260810 23:00:27.824062 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 23:00:27.824450 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 18489
I20260810 23:00:27.824576 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 23:00:27.824854 30010 external_mini_cluster.cc:1428] Setting key ce54a66d9bfe344e21cab14f16c0a152
I20260810 23:00:27.827064 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36039
--builtin_ntp_servers=127.29.78.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:00:27.833892 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.825756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:39515 (local address 127.29.78.190:36039)
0810 23:00:27.825905 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:27.825909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:27.827176 (+ 1267us) server_negotiation.cc:407] Connection header received
0810 23:00:27.829040 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:27.829045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:27.829101 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:27.829191 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:27.830124 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.830844 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.831960 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.832198 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.832309 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.833391 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:27.833394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:27.833396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:27.833407 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:27.833430 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:27.833442 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:27.833508 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:27.833649 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 23:00:27.833725 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:00:27.834545 18636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:27.826243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36039 (local address 127.29.78.129:39515)
0810 23:00:27.827003 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:27.827055 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:27.828818 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:27.829327 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:27.829332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:27.829400 (+ 68us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:27.829978 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.829998 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.830963 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.830966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.831789 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:27.831797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:27.832345 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:27.832349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:27.832512 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:27.833155 (+ 643us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:27.833156 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:27.833189 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:27.833241 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:27.833520 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:27.833529 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:27.833534 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:00:27.833648 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:00:27.833886 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":143,"threads_started":1}
I20260810 23:00:27.835629 18634 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36039
I20260810 23:00:27.835881 18634 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:27.836355 18634 heartbeater.cc:511] Master 127.29.78.190:36039 requested a full tablet report, sending...
I20260810 23:00:27.837934 18420 ts_manager.cc:194] Registered new tserver with Master: 1b06ce702bc141f99bd10f9c1960a0c4 (127.29.78.129:36431)
I20260810 23:00:27.839558 18420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:39515
W20260810 23:00:27.934067 18637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:27.934288 18637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:27.934322 18637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:27.938089 18637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:27.938127 18637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:27.938170 18637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 23:00:27.942194 18637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:27.943113 18637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:27.944284 18637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:27.953562 18645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:27.953620 18644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:27.953625 18647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:27.953706 18637 server_base.cc:1034] running on GCE node
I20260810 23:00:27.954218 18637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:27.954783 18637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:27.955966 18637 hybrid_clock.cc:648] HybridClock initialized: now 1786402827955953 us; error 38 us; skew 500 ppm
I20260810 23:00:27.957685 18637 webserver.cc:460] Webserver started at http://127.29.78.130:37973/ using document root <none> and password file <none>
I20260810 23:00:27.958256 18637 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:27.958343 18637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:27.958575 18637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:27.960490 18637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "5bed6dace81c4a8dab2ec91b86fa09df"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "fc3e7ed5c28e55455a9c362dd3a9a5bc"
server_key_iv: "a637906bcb1c7120b1048941da552f26"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.961020 18637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "5bed6dace81c4a8dab2ec91b86fa09df"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "fc3e7ed5c28e55455a9c362dd3a9a5bc"
server_key_iv: "a637906bcb1c7120b1048941da552f26"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.964262 18637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 23:00:27.966413 18655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.967582 18637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:27.967715 18637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "5bed6dace81c4a8dab2ec91b86fa09df"
format_stamp: "Formatted at 2026-08-10 23:00:27 on dist-test-slave-6nmv"
server_key: "fc3e7ed5c28e55455a9c362dd3a9a5bc"
server_key_iv: "a637906bcb1c7120b1048941da552f26"
server_key_version: "encryptionkey@0"
I20260810 23:00:27.967813 18637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:27.989355 18637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:27.990075 18637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:27.990253 18637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:27.990775 18637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:27.991722 18637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:27.991792 18637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.991859 18637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:27.991909 18637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:27.999890 18637 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:34937
I20260810 23:00:27.999910 18782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:34937 every 8 connection(s)
I20260810 23:00:28.000715 18637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 23:00:28.004694 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 18637
I20260810 23:00:28.004846 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 23:00:28.005177 30010 external_mini_cluster.cc:1428] Setting key d61454ffe8a47f6f70b61c07f9838f96
I20260810 23:00:28.007259 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36039
--builtin_ntp_servers=127.29.78.148:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:00:28.010663 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.002432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:52029 (local address 127.29.78.190:36039)
0810 23:00:28.002591 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.002596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.003657 (+ 1061us) server_negotiation.cc:407] Connection header received
0810 23:00:28.005233 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.005238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.005301 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.005364 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:28.006403 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.007040 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.008667 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.008872 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.008985 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.010161 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:28.010164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:28.010167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:28.010186 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:28.010203 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:28.010211 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:28.010289 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.010454 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.010522 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:00:28.011431 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.002806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36039 (local address 127.29.78.130:52029)
0810 23:00:28.003514 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.003552 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:28.004985 (+ 1433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:28.005560 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:28.005568 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:28.005658 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:28.006215 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.006229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.007180 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.007184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.008539 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.008554 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.009036 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.009040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.009223 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.009884 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:28.009886 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:28.009945 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:28.010006 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:28.010292 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:28.010304 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:28.010312 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:00:28.010444 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 23:00:28.010704 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":108,"threads_started":1}
I20260810 23:00:28.012717 18783 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36039
I20260810 23:00:28.012991 18783 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:28.013501 18783 heartbeater.cc:511] Master 127.29.78.190:36039 requested a full tablet report, sending...
I20260810 23:00:28.014576 18420 ts_manager.cc:194] Registered new tserver with Master: 5bed6dace81c4a8dab2ec91b86fa09df (127.29.78.130:34937)
I20260810 23:00:28.015182 18420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:52029
W20260810 23:00:28.111618 18786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:28.111867 18786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:28.111931 18786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:28.115938 18786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:28.115998 18786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:28.116066 18786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 23:00:28.120359 18786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:36039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:28.121196 18786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:28.122336 18786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:28.130766 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:28.130852 18786 server_base.cc:1034] running on GCE node
W20260810 23:00:28.130846 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:28.130846 18794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:28.131412 18786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:28.131932 18786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:28.133119 18786 hybrid_clock.cc:648] HybridClock initialized: now 1786402828133082 us; error 31 us; skew 500 ppm
I20260810 23:00:28.134697 18786 webserver.cc:460] Webserver started at http://127.29.78.131:38593/ using document root <none> and password file <none>
I20260810 23:00:28.135167 18786 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:28.135219 18786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:28.135416 18786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:28.137223 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "ba8657dbc54341609582e75c78629c85"
format_stamp: "Formatted at 2026-08-10 23:00:28 on dist-test-slave-6nmv"
server_key: "bef4fcf14ca11cbe899f5d6ab5bdf539"
server_key_iv: "abda44bde29501891a43e8bcf05983aa"
server_key_version: "encryptionkey@0"
I20260810 23:00:28.137759 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "ba8657dbc54341609582e75c78629c85"
format_stamp: "Formatted at 2026-08-10 23:00:28 on dist-test-slave-6nmv"
server_key: "bef4fcf14ca11cbe899f5d6ab5bdf539"
server_key_iv: "abda44bde29501891a43e8bcf05983aa"
server_key_version: "encryptionkey@0"
I20260810 23:00:28.140898 18786 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:00:28.142928 18804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:28.143963 18786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:28.144093 18786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "ba8657dbc54341609582e75c78629c85"
format_stamp: "Formatted at 2026-08-10 23:00:28 on dist-test-slave-6nmv"
server_key: "bef4fcf14ca11cbe899f5d6ab5bdf539"
server_key_iv: "abda44bde29501891a43e8bcf05983aa"
server_key_version: "encryptionkey@0"
I20260810 23:00:28.144191 18786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:28.157195 18786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:28.157753 18786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:28.157955 18786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:28.158440 18786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:28.159286 18786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:28.159356 18786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:28.159431 18786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:28.159483 18786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:28.167584 18786 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:33251
I20260810 23:00:28.167622 18930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:33251 every 8 connection(s)
I20260810 23:00:28.168406 18786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 23:00:28.173790 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 18786
I20260810 23:00:28.173921 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 23:00:28.174219 30010 external_mini_cluster.cc:1428] Setting key 94ded6db668b3694a3b577409f97df13
I20260810 23:00:28.177356 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.170050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:45209 (local address 127.29.78.190:36039)
0810 23:00:28.170209 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.170213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.171221 (+ 1008us) server_negotiation.cc:407] Connection header received
0810 23:00:28.172511 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.172513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.172559 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.172631 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:28.173469 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.174118 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.175549 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.175834 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.175932 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.176957 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:28.176968 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:28.176970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:28.176981 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:28.177000 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:28.177008 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:28.177067 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.177202 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.177250 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:00:28.178124 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.170449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:36039 (local address 127.29.78.131:45209)
0810 23:00:28.171041 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.171077 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:28.172315 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:28.172790 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:28.172807 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:28.172879 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:28.173346 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.173357 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.174236 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.174241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.175434 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.175443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.175969 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.175972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.176186 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.176728 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:28.176730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:28.176765 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:28.176822 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:28.177082 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:28.177089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:28.177092 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:00:28.177178 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:00:28.177365 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
I20260810 23:00:28.179072 18931 heartbeater.cc:347] Connected to a master server at 127.29.78.190:36039
I20260810 23:00:28.179287 18931 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:28.179720 18931 heartbeater.cc:511] Master 127.29.78.190:36039 requested a full tablet report, sending...
I20260810 23:00:28.180598 18420 ts_manager.cc:194] Registered new tserver with Master: ba8657dbc54341609582e75c78629c85 (127.29.78.131:33251)
I20260810 23:00:28.181115 18420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:45209
I20260810 23:00:28.187674 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:00:28.193027 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.189424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.29.78.190:36039)
0810 23:00:28.189546 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.189550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.189714 (+ 164us) server_negotiation.cc:407] Connection header received
0810 23:00:28.189873 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.189876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.189913 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.189997 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:28.190245 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.190689 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.191459 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.191695 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.191796 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.192524 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:28.192528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:28.192531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:28.192543 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:28.192562 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:28.192575 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:28.192699 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.192852 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.192900 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:00:28.195560 18420 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:00:28.198379 18420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:00:28.211596 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.206505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.29.78.131:33251)
0810 23:00:28.207229 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.207242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.207267 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:00:28.207333 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.207338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.207505 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.207643 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:28.207933 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.208567 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.210371 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.210937 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.211060 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.211235 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.211355 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.211439 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":125,"threads_started":1}
I20260810 23:00:28.211578 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.206190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33251 (local address 127.0.0.1:59584)
0810 23:00:28.206663 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.206678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:28.206767 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:28.207699 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:28.207702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:28.207718 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:28.207813 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.207818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.208679 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.208682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.210239 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.210250 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.211061 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.211065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.211169 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.211207 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:00:28.211328 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:00:28.211446 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":267,"thread_start_us":155,"threads_started":1}
I20260810 23:00:28.212157 18952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.206508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36431 (local address 127.0.0.1:36898)
0810 23:00:28.206883 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.206895 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:28.206999 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:28.207781 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:28.207784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:28.207811 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:28.207893 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.207899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.209215 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.209220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.210369 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.210380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.211576 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.211579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.211778 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.211854 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 23:00:28.211924 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:00:28.212025 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":176,"thread_start_us":93,"threads_started":1}
I20260810 23:00:28.212265 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.206934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.29.78.129:36431)
0810 23:00:28.207369 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.207379 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.207397 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:00:28.207477 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.207481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.207642 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.208042 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:28.208166 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.209115 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.210508 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.211247 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.211347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.211508 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.211991 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.212099 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":153,"threads_started":1}
I20260810 23:00:28.212371 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.205996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.29.78.130:34937)
0810 23:00:28.207019 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.207032 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.207057 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:00:28.207139 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.207147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.207350 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.207547 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:28.207802 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.208866 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.210401 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.211207 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.211334 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.211508 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.212101 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.212207 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":112,"threads_started":1}
I20260810 23:00:28.212697 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.205786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:34937 (local address 127.0.0.1:42876)
0810 23:00:28.206140 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.206184 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:28.206313 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:28.207535 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:28.207540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:28.207561 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:28.207691 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.207697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.208991 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.208994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.210265 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.210273 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.211576 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.211579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.211787 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.211849 (+ 62us) client_negotiation.cc:769] Sending connection context
0810 23:00:28.211923 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:00:28.212025 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":248,"thread_start_us":131,"threads_started":1}
I20260810 23:00:28.215029 18561 tablet_service.cc:1467] Processing CreateTablet for tablet a95ee78d46f943b89b82dd1b198e4c10 (DEFAULT_TABLE table=test-table [id=fd5f23252d654668971033bfa4b16758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:28.215130 18858 tablet_service.cc:1467] Processing CreateTablet for tablet a95ee78d46f943b89b82dd1b198e4c10 (DEFAULT_TABLE table=test-table [id=fd5f23252d654668971033bfa4b16758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:28.215493 18712 tablet_service.cc:1467] Processing CreateTablet for tablet a95ee78d46f943b89b82dd1b198e4c10 (DEFAULT_TABLE table=test-table [id=fd5f23252d654668971033bfa4b16758]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:28.215974 18561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95ee78d46f943b89b82dd1b198e4c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:28.216094 18858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95ee78d46f943b89b82dd1b198e4c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:28.216393 18712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a95ee78d46f943b89b82dd1b198e4c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:28.221680 18956 tablet_bootstrap.cc:492] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4: Bootstrap starting.
I20260810 23:00:28.223472 18958 tablet_bootstrap.cc:492] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Bootstrap starting.
I20260810 23:00:28.223801 18957 tablet_bootstrap.cc:492] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Bootstrap starting.
I20260810 23:00:28.224202 18956 tablet_bootstrap.cc:654] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:28.224870 18956 log.cc:824] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:28.225617 18957 tablet_bootstrap.cc:654] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:28.225617 18958 tablet_bootstrap.cc:654] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:28.226518 18958 log.cc:824] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:28.226518 18957 log.cc:824] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:28.226569 18956 tablet_bootstrap.cc:492] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4: No bootstrap required, opened a new log
I20260810 23:00:28.226768 18956 ts_tablet_manager.cc:1397] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 23:00:28.228438 18957 tablet_bootstrap.cc:492] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: No bootstrap required, opened a new log
I20260810 23:00:28.228677 18957 ts_tablet_manager.cc:1397] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 23:00:28.228852 18958 tablet_bootstrap.cc:492] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: No bootstrap required, opened a new log
I20260810 23:00:28.229072 18958 ts_tablet_manager.cc:1397] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:00:28.230661 18956 raft_consensus.cc:348] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.230890 18956 raft_consensus.cc:374] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:28.230964 18956 raft_consensus.cc:729] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b06ce702bc141f99bd10f9c1960a0c4, State: Initialized, Role: FOLLOWER
I20260810 23:00:28.231396 18956 consensus_queue.cc:260] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.232038 18634 heartbeater.cc:503] Master 127.29.78.190:36039 was elected leader, sending a full tablet report...
I20260810 23:00:28.232371 18956 ts_tablet_manager.cc:1428] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:00:28.233258 18958 raft_consensus.cc:348] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.233258 18957 raft_consensus.cc:348] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.233543 18957 raft_consensus.cc:374] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:28.233655 18957 raft_consensus.cc:729] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8657dbc54341609582e75c78629c85, State: Initialized, Role: FOLLOWER
I20260810 23:00:28.233704 18958 raft_consensus.cc:374] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:28.233798 18958 raft_consensus.cc:729] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bed6dace81c4a8dab2ec91b86fa09df, State: Initialized, Role: FOLLOWER
I20260810 23:00:28.234195 18957 consensus_queue.cc:260] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.234195 18958 consensus_queue.cc:260] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.235121 18957 ts_tablet_manager.cc:1428] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:00:28.235307 18783 heartbeater.cc:503] Master 127.29.78.190:36039 was elected leader, sending a full tablet report...
I20260810 23:00:28.235313 18958 ts_tablet_manager.cc:1428] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260810 23:00:28.235185 18931 heartbeater.cc:503] Master 127.29.78.190:36039 was elected leader, sending a full tablet report...
W20260810 23:00:28.251578 18784 tablet.cc:2369] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:00:28.325341 18635 tablet.cc:2369] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:28.379078 18964 raft_consensus.cc:482] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:28.379303 18964 raft_consensus.cc:504] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.380579 18964 leader_election.cc:288] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b06ce702bc141f99bd10f9c1960a0c4 (127.29.78.129:36431), ba8657dbc54341609582e75c78629c85 (127.29.78.131:33251)
I20260810 23:00:28.384261 18954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.381014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:49461 (local address 127.29.78.129:36431)
0810 23:00:28.381148 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.381153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.381175 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:00:28.381222 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.381224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.381263 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.381353 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:28.381648 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.382269 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.383311 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.383706 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.383853 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.383906 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.384076 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.384143 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:00:28.384261 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.380779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36431 (local address 127.29.78.130:49461)
0810 23:00:28.380923 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.380938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:28.381058 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:28.381366 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:28.381369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:28.381386 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:28.381521 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.381526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.382376 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.382381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.383179 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.383188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.383841 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.383845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.383940 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.383962 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:00:28.384060 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:00:28.384130 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":24}
I20260810 23:00:28.384406 18953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.380981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:55587 (local address 127.29.78.131:33251)
0810 23:00:28.381147 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.381152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.381367 (+ 215us) server_negotiation.cc:407] Connection header received
0810 23:00:28.381532 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.381535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.381583 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.381666 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:28.381969 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.382458 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.383461 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.383856 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.383988 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.384034 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.384213 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.384276 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:00:28.384423 18966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.380861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:33251 (local address 127.29.78.130:55587)
0810 23:00:28.381278 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.381289 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:28.381407 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:28.381676 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:28.381678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:28.381694 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:28.381859 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.381866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.382576 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.382580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.383334 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:28.383342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.383988 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.383992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:28.384086 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.384113 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:00:28.384198 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:00:28.384262 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":2,"thread_start_us":148,"threads_started":1}
I20260810 23:00:28.384979 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95ee78d46f943b89b82dd1b198e4c10" candidate_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" is_pre_election: true
I20260810 23:00:28.384986 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95ee78d46f943b89b82dd1b198e4c10" candidate_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8657dbc54341609582e75c78629c85" is_pre_election: true
I20260810 23:00:28.385263 18878 raft_consensus.cc:2454] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bed6dace81c4a8dab2ec91b86fa09df in term 0.
I20260810 23:00:28.385267 18586 raft_consensus.cc:2454] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bed6dace81c4a8dab2ec91b86fa09df in term 0.
I20260810 23:00:28.385753 18657 leader_election.cc:302] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bed6dace81c4a8dab2ec91b86fa09df, ba8657dbc54341609582e75c78629c85; no voters:
I20260810 23:00:28.386063 18964 raft_consensus.cc:2781] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:00:28.386148 18964 raft_consensus.cc:482] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:00:28.386180 18964 raft_consensus.cc:3037] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:28.387235 18964 raft_consensus.cc:504] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.387635 18964 leader_election.cc:288] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [CANDIDATE]: Term 1 election: Requested vote from peers 1b06ce702bc141f99bd10f9c1960a0c4 (127.29.78.129:36431), ba8657dbc54341609582e75c78629c85 (127.29.78.131:33251)
I20260810 23:00:28.387900 18586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95ee78d46f943b89b82dd1b198e4c10" candidate_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b06ce702bc141f99bd10f9c1960a0c4"
I20260810 23:00:28.387977 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a95ee78d46f943b89b82dd1b198e4c10" candidate_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8657dbc54341609582e75c78629c85"
I20260810 23:00:28.388036 18586 raft_consensus.cc:3037] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:28.388116 18878 raft_consensus.cc:3037] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:28.389030 18586 raft_consensus.cc:2454] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bed6dace81c4a8dab2ec91b86fa09df in term 1.
I20260810 23:00:28.389117 18878 raft_consensus.cc:2454] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bed6dace81c4a8dab2ec91b86fa09df in term 1.
I20260810 23:00:28.389393 18658 leader_election.cc:302] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b06ce702bc141f99bd10f9c1960a0c4, 5bed6dace81c4a8dab2ec91b86fa09df; no voters:
I20260810 23:00:28.389649 18964 raft_consensus.cc:2781] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:28.389918 18964 raft_consensus.cc:686] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 1 LEADER]: Becoming Leader. State: Replica: 5bed6dace81c4a8dab2ec91b86fa09df, State: Running, Role: LEADER
I20260810 23:00:28.390264 18964 consensus_queue.cc:237] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } }
I20260810 23:00:28.393415 18419 catalog_manager.cc:5495] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bed6dace81c4a8dab2ec91b86fa09df (127.29.78.130). New cstate: current_term: 1 leader_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5bed6dace81c4a8dab2ec91b86fa09df" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 34937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 } health_report { overall_health: UNKNOWN } } }
W20260810 23:00:28.419318 18933 tablet.cc:2369] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:28.445080 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.441688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.29.78.130:34937)
0810 23:00:28.441836 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:28.441840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:28.441854 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:00:28.441963 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:28.441965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:28.442016 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:28.442121 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:00:28.442372 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.442812 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.443645 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:28.443879 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:28.444009 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:28.444291 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:00:28.444651 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:00:28.444800 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:28.444896 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 23:00:28.444929 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:00:28.450662 18586 raft_consensus.cc:1261] T a95ee78d46f943b89b82dd1b198e4c10 P 1b06ce702bc141f99bd10f9c1960a0c4 [term 1 FOLLOWER]: Refusing update from remote peer 5bed6dace81c4a8dab2ec91b86fa09df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:28.450661 18878 raft_consensus.cc:1261] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 1 FOLLOWER]: Refusing update from remote peer 5bed6dace81c4a8dab2ec91b86fa09df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:28.451380 18967 consensus_queue.cc:1035] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8657dbc54341609582e75c78629c85" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:28.451613 18964 consensus_queue.cc:1035] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b06ce702bc141f99bd10f9c1960a0c4" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 36431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:28.460876 18972 mvcc.cc:205] Tried to move back new op lower bound from 7317105985326948352 to 7317105985089069056. Current Snapshot: MvccSnapshot[applied={T|T < 7317105985326948352 or (T in {7317105985326948352})}]
I20260810 23:00:28.466244 18419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:28.466432 18419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:28.468672 18419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bef78c118004eadb8bc994060c7701e: Sending DeleteTablet for 3 replicas of tablet a95ee78d46f943b89b82dd1b198e4c10
I20260810 23:00:28.469405 18712 tablet_service.cc:1514] Processing DeleteTablet for tablet a95ee78d46f943b89b82dd1b198e4c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:28 UTC) from {username='slave'} at 127.0.0.1:42876
I20260810 23:00:28.469791 18980 tablet_replica.cc:330] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: stopping tablet replica
I20260810 23:00:28.469789 18561 tablet_service.cc:1514] Processing DeleteTablet for tablet a95ee78d46f943b89b82dd1b198e4c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:28 UTC) from {username='slave'} at 127.0.0.1:36898
I20260810 23:00:28.469964 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 18489
I20260810 23:00:28.470150 18980 raft_consensus.cc:2229] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:00:28.470461 18980 raft_consensus.cc:2258] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:28.474839 18858 tablet_service.cc:1514] Processing DeleteTablet for tablet a95ee78d46f943b89b82dd1b198e4c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:28 UTC) from {username='slave'} at 127.0.0.1:59584
I20260810 23:00:28.475252 18980 ts_tablet_manager.cc:1905] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:28.475535 18981 tablet_replica.cc:330] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: stopping tablet replica
I20260810 23:00:28.475837 18981 raft_consensus.cc:2229] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:28.476122 18981 raft_consensus.cc:2258] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:28.477512 18981 ts_tablet_manager.cc:1905] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:28.478726 18980 ts_tablet_manager.cc:1918] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:28.478878 18980 log.cc:1196] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/a95ee78d46f943b89b82dd1b198e4c10
I20260810 23:00:28.479274 18980 ts_tablet_manager.cc:1939] T a95ee78d46f943b89b82dd1b198e4c10 P 5bed6dace81c4a8dab2ec91b86fa09df: Deleting consensus metadata
I20260810 23:00:28.479988 18981 ts_tablet_manager.cc:1918] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:28.480119 18981 log.cc:1196] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/a95ee78d46f943b89b82dd1b198e4c10
I20260810 23:00:28.480414 18981 ts_tablet_manager.cc:1939] T a95ee78d46f943b89b82dd1b198e4c10 P ba8657dbc54341609582e75c78629c85: Deleting consensus metadata
I20260810 23:00:28.480476 18403 catalog_manager.cc:4841] TS 5bed6dace81c4a8dab2ec91b86fa09df (127.29.78.130:34937): tablet a95ee78d46f943b89b82dd1b198e4c10 (table test-table [id=fd5f23252d654668971033bfa4b16758]) successfully deleted
I20260810 23:00:28.481387 18401 catalog_manager.cc:4841] TS ba8657dbc54341609582e75c78629c85 (127.29.78.131:33251): tablet a95ee78d46f943b89b82dd1b198e4c10 (table test-table [id=fd5f23252d654668971033bfa4b16758]) successfully deleted
W20260810 23:00:28.482384 18403 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:36431 (error 108)
I20260810 23:00:28.482733 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 18637
I20260810 23:00:28.490684 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:28.490458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:36431 (local address 127.0.0.1:36904)
0810 23:00:28.490566 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:28.490621 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:36431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:00:28.490916 18403 catalog_manager.cc:4584] TS 1b06ce702bc141f99bd10f9c1960a0c4 (127.29.78.129:36431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a95ee78d46f943b89b82dd1b198e4c10: Network error: Client connection negotiation failed: client connection to 127.29.78.129:36431: connect: Connection refused (error 111)
I20260810 23:00:28.493119 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 18786
I20260810 23:00:28.499817 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 18383
2026-08-10T23:00:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1101 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:28 dist-test-slave-6nmv krb5kdc[18986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:28 dist-test-slave-6nmv krb5kdc[18986](info): set up 2 sockets
Aug 10 23:00:28 dist-test-slave-6nmv krb5kdc[18986](info): commencing operation
krb5kdc: starting...
W20260810 23:00:29.543351 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:29 dist-test-slave-6nmv krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:29.654728 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40561
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:29.765516 18999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:29.765770 18999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:29.765854 18999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:29.770681 18999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:29.770746 18999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:29.770778 18999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:29.770808 18999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:29.770838 18999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:29.778101 18999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:29.779083 18999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:29.780117 18999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:29.790016 19005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:29.790526 19004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:29.790488 19007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:29.793915 18999 server_base.cc:1034] running on GCE node
I20260810 23:00:29.794471 18999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:29.795655 18999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:29.796895 18999 hybrid_clock.cc:648] HybridClock initialized: now 1786402829796854 us; error 44 us; skew 500 ppm
Aug 10 23:00:29 dist-test-slave-6nmv krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402829, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:29.800611 18999 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:29.801579 18999 webserver.cc:460] Webserver started at http://127.29.78.190:43505/ using document root <none> and password file <none>
I20260810 23:00:29.802206 18999 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:29.802263 18999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:29.802466 18999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:29.804438 18999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "e38b1cb70bed4e5c92399ff8bc12d2be"
format_stamp: "Formatted at 2026-08-10 23:00:29 on dist-test-slave-6nmv"
server_key: "a0ef25fb9f55ca4b3dbe386c2981090f"
server_key_iv: "478661725cddc9b99595fbd0f3029e91"
server_key_version: "encryptionkey@0"
I20260810 23:00:29.805018 18999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "e38b1cb70bed4e5c92399ff8bc12d2be"
format_stamp: "Formatted at 2026-08-10 23:00:29 on dist-test-slave-6nmv"
server_key: "a0ef25fb9f55ca4b3dbe386c2981090f"
server_key_iv: "478661725cddc9b99595fbd0f3029e91"
server_key_version: "encryptionkey@0"
I20260810 23:00:29.808779 18999 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:00:29.811304 19014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:29.812577 18999 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:29.812719 18999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "e38b1cb70bed4e5c92399ff8bc12d2be"
format_stamp: "Formatted at 2026-08-10 23:00:29 on dist-test-slave-6nmv"
server_key: "a0ef25fb9f55ca4b3dbe386c2981090f"
server_key_iv: "478661725cddc9b99595fbd0f3029e91"
server_key_version: "encryptionkey@0"
I20260810 23:00:29.812830 18999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:29.831384 18999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:29.834213 18999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:29.834421 18999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:29.841583 18999 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40561
I20260810 23:00:29.841609 19072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40561 every 8 connection(s)
I20260810 23:00:29.842474 18999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:29.845108 19073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:29.850378 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 18999
I20260810 23:00:29.850383 19073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be: Bootstrap starting.
I20260810 23:00:29.850520 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:29.850793 30010 external_mini_cluster.cc:1428] Setting key 8ac50fd1b57fe0611794124603ab2325
I20260810 23:00:29.853080 19073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:29.853957 19073 log.cc:824] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:29.856099 19073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be: No bootstrap required, opened a new log
Aug 10 23:00:29 dist-test-slave-6nmv krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:29.860070 19073 raft_consensus.cc:348] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40561 } }
I20260810 23:00:29.860308 19073 raft_consensus.cc:374] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:29.860394 19073 raft_consensus.cc:729] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38b1cb70bed4e5c92399ff8bc12d2be, State: Initialized, Role: FOLLOWER
I20260810 23:00:29.860867 19073 consensus_queue.cc:260] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40561 } }
I20260810 23:00:29.861032 19073 raft_consensus.cc:388] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:29.861141 19073 raft_consensus.cc:482] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:29.861245 19073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:29.862248 19073 raft_consensus.cc:504] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40561 } }
I20260810 23:00:29.862581 19073 leader_election.cc:302] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e38b1cb70bed4e5c92399ff8bc12d2be; no voters:
I20260810 23:00:29.862908 19073 leader_election.cc:288] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:29.862991 19079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:29.863232 19079 raft_consensus.cc:686] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [term 1 LEADER]: Becoming Leader. State: Replica: e38b1cb70bed4e5c92399ff8bc12d2be, State: Running, Role: LEADER
I20260810 23:00:29.863590 19079 consensus_queue.cc:237] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40561 } }
I20260810 23:00:29.864099 19073 sys_catalog.cc:564] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:29.866634 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:29.852187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.29.78.190:40561)
0810 23:00:29.852579 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:29.852597 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:29.852637 (+ 40us) server_negotiation.cc:407] Connection header received
0810 23:00:29.853517 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:29.853539 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:29.853980 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:29.854334 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:29.855165 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:29.856584 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:29.857418 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:29.858151 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:29.858328 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:29.860784 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:29.860789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:29.860800 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:29.860825 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:29.862606 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:29.863128 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:29.863134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:29.863139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:29.863283 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:29.863603 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:29.863609 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:29.863611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:29.864085 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:29.864250 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:29.864792 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 23:00:29.865126 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260810 23:00:29.866642 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40561 } } }
I20260810 23:00:29.866662 19082 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38b1cb70bed4e5c92399ff8bc12d2be. Latest consensus state: current_term: 1 leader_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e38b1cb70bed4e5c92399ff8bc12d2be" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40561 } } }
I20260810 23:00:29.866887 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:29.866887 19082 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be [sys.catalog]: This master's current role is: LEADER
W20260810 23:00:29.869447 19096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:00:29.869541 19096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:00:29.869604 19098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:29.870383 19098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:29.875274 19098 catalog_manager.cc:1347] Generated new cluster ID: 8728b0e7789c4aaaa3fcc09c2f21a0f0
I20260810 23:00:29.875336 19098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:29.890156 19098 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:29.890861 19098 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:29.907676 19098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be: Generated new TSK 0
I20260810 23:00:29.908288 19098 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.29.78.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
Entry for principal kudu/127.29.78.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab.
I20260810 23:00:29.940747 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40561
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:30.049847 19103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:30.050159 19103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:30.050237 19103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:30.054311 19103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:30.054394 19103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:30.054486 19103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 23:00:30.059759 19103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.129.keytab
--principal=kudu/127.29.78.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:30.060827 19103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:30.061980 19103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:30.072455 19113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:30.072455 19111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:30.072472 19110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:30.072778 19103 server_base.cc:1034] running on GCE node
I20260810 23:00:30.073189 19103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:30.073963 19103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:30.075150 19103 hybrid_clock.cc:648] HybridClock initialized: now 1786402830075138 us; error 38 us; skew 500 ppm
Aug 10 23:00:30 dist-test-slave-6nmv krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:30.078097 19103 init.cc:377] Logged in from keytab as kudu/127.29.78.129@KRBTEST.COM (short username kudu)
I20260810 23:00:30.079068 19103 webserver.cc:460] Webserver started at http://127.29.78.129:35677/ using document root <none> and password file <none>
I20260810 23:00:30.079655 19103 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:30.079751 19103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:30.079998 19103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:30.082130 19103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "7ea2973098f74ad1bd3792e990363d08"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "b84314455ae64d4dd9681c2518535195"
server_key_iv: "b2bb403d2e8d80feefe42cf508dfce10"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.082715 19103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "7ea2973098f74ad1bd3792e990363d08"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "b84314455ae64d4dd9681c2518535195"
server_key_iv: "b2bb403d2e8d80feefe42cf508dfce10"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.086530 19103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 23:00:30.088927 19121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.090198 19103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:30.090315 19103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "7ea2973098f74ad1bd3792e990363d08"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "b84314455ae64d4dd9681c2518535195"
server_key_iv: "b2bb403d2e8d80feefe42cf508dfce10"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.090415 19103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:30.123728 19103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:30.127185 19103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:30.127388 19103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:30.127996 19103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:30.128814 19103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:30.128861 19103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.128901 19103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:30.128955 19103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.137494 19103 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:38943
I20260810 23:00:30.137557 19249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:38943 every 8 connection(s)
I20260810 23:00:30.138458 19103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 23:00:30.146898 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 19103
I20260810 23:00:30.147055 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 23:00:30.147408 30010 external_mini_cluster.cc:1428] Setting key 92693e6f70cc6767f342360f32797bbf
Aug 10 23:00:30 dist-test-slave-6nmv krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.129@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:30.152683 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.140251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:53635 (local address 127.29.78.190:40561)
0810 23:00:30.140402 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.140406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.141176 (+ 770us) server_negotiation.cc:407] Connection header received
0810 23:00:30.142556 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.142560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.142620 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.142744 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:30.143845 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.144423 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.145564 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.145848 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.145982 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.148780 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:30.148784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:30.148787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:30.148817 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:30.150614 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.151129 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.151133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.151135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.151223 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.151578 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.151582 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.151598 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.151781 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:30.151891 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.152365 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.152508 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:00:30.153455 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.140588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40561 (local address 127.29.78.129:53635)
0810 23:00:30.141022 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.141064 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.142331 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.142909 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.142916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.143230 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:30.143671 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.143684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.144556 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.144560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.145422 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.145431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.146009 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.146014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.146182 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.146733 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:30.146753 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:30.148607 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:30.150741 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:30.150746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:30.150756 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:30.151016 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:30.151329 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:30.151337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:30.151338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:30.151478 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:30.151889 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:30.151896 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:30.152135 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.152323 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.152645 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20260810 23:00:30.154845 19250 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40561
I20260810 23:00:30.155133 19250 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:30.155747 19250 heartbeater.cc:511] Master 127.29.78.190:40561 requested a full tablet report, sending...
I20260810 23:00:30.157313 19034 ts_manager.cc:194] Registered new tserver with Master: 7ea2973098f74ad1bd3792e990363d08 (127.29.78.129:38943)
WARNING: no policy specified for kudu/127.29.78.130@KRBTEST.COM; defaulting to no policy
I20260810 23:00:30.158529 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.129@KRBTEST.COM'} at 127.29.78.129:53635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
Entry for principal kudu/127.29.78.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab.
I20260810 23:00:30.174679 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40561
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:30.280490 19257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:30.280712 19257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:30.280751 19257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:30.284320 19257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:30.284361 19257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:30.284411 19257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 23:00:30.289011 19257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.130.keytab
--principal=kudu/127.29.78.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:30.289918 19257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:30.290721 19257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:30.299746 19263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:30.299770 19266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:30.299770 19264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:30.300029 19257 server_base.cc:1034] running on GCE node
I20260810 23:00:30.300379 19257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:30.300906 19257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:30.302071 19257 hybrid_clock.cc:648] HybridClock initialized: now 1786402830302047 us; error 35 us; skew 500 ppm
Aug 10 23:00:30 dist-test-slave-6nmv krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:30.308851 19257 init.cc:377] Logged in from keytab as kudu/127.29.78.130@KRBTEST.COM (short username kudu)
I20260810 23:00:30.309652 19257 webserver.cc:460] Webserver started at http://127.29.78.130:39093/ using document root <none> and password file <none>
I20260810 23:00:30.310207 19257 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:30.310292 19257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:30.310505 19257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:30.312350 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "7ddfeead292a4658bfc80b90be34f325"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "fc40b4f337161425040bab0e35c4035f"
server_key_iv: "67c06d07b47feef7cf2fd93e09e13e92"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.312919 19257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "7ddfeead292a4658bfc80b90be34f325"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "fc40b4f337161425040bab0e35c4035f"
server_key_iv: "67c06d07b47feef7cf2fd93e09e13e92"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.316466 19257 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 23:00:30.318764 19273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.319839 19257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:30.319970 19257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "7ddfeead292a4658bfc80b90be34f325"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "fc40b4f337161425040bab0e35c4035f"
server_key_iv: "67c06d07b47feef7cf2fd93e09e13e92"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.320071 19257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:30.336228 19257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:30.339296 19257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:30.339498 19257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:30.340027 19257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:30.340776 19257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:30.340849 19257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.340934 19257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:30.340976 19257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.349282 19257 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:43187
I20260810 23:00:30.349310 19403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:43187 every 8 connection(s)
I20260810 23:00:30.350142 19257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 23:00:30.350435 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 19257
I20260810 23:00:30.350611 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 23:00:30.351030 30010 external_mini_cluster.cc:1428] Setting key d66a9ed91d3c3e0f2e2181241fee2975
Aug 10 23:00:30 dist-test-slave-6nmv krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.130@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.131@KRBTEST.COM; defaulting to no policy
I20260810 23:00:30.363895 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.352001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:35323 (local address 127.29.78.190:40561)
0810 23:00:30.352136 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.352141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.353058 (+ 917us) server_negotiation.cc:407] Connection header received
0810 23:00:30.354356 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.354359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.354412 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.354514 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:30.355684 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.356163 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.357274 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.357495 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.357603 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.360384 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:30.360387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:30.360389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:30.360411 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:30.361949 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.362505 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.362508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.362509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.362581 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.362919 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.362921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.362922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.363087 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:30.363157 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.363690 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.363786 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.131@KRBTEST.COM" created.
I20260810 23:00:30.364923 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.352413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40561 (local address 127.29.78.130:35323)
0810 23:00:30.352925 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.352969 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.354166 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.354669 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.354675 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.355082 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:30.355538 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.355551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.356284 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.356287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.357148 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.357158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.357640 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.357645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.357878 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.358421 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:30.358443 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:30.360216 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:30.362054 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:30.362058 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:30.362068 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:30.362401 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:30.362707 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:30.362709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:30.362711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:30.362836 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:30.363172 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:30.363176 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:30.363479 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.363640 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.363896 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":177,"threads_started":1}
I20260810 23:00:30.366276 19404 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40561
I20260810 23:00:30.366554 19404 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:30.367045 19404 heartbeater.cc:511] Master 127.29.78.190:40561 requested a full tablet report, sending...
I20260810 23:00:30.368139 19034 ts_manager.cc:194] Registered new tserver with Master: 7ddfeead292a4658bfc80b90be34f325 (127.29.78.130:43187)
I20260810 23:00:30.368984 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.130@KRBTEST.COM'} at 127.29.78.130:35323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
Entry for principal kudu/127.29.78.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab.
I20260810 23:00:30.378340 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40561
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:43227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:30.483213 19409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:30.483424 19409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:30.483469 19409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:30.487174 19409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:30.487218 19409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:30.487272 19409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 23:00:30.492240 19409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:43227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.131.keytab
--principal=kudu/127.29.78.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:40561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:30.493201 19409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:30.494201 19409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:30.504065 19415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:30.504072 19417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:30.504088 19414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:30.504235 19409 server_base.cc:1034] running on GCE node
I20260810 23:00:30.504745 19409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:30.505316 19409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:30.506469 19409 hybrid_clock.cc:648] HybridClock initialized: now 1786402830506446 us; error 26 us; skew 500 ppm
Aug 10 23:00:30 dist-test-slave-6nmv krb5kdc[18986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:30.509064 19409 init.cc:377] Logged in from keytab as kudu/127.29.78.131@KRBTEST.COM (short username kudu)
I20260810 23:00:30.509851 19409 webserver.cc:460] Webserver started at http://127.29.78.131:36003/ using document root <none> and password file <none>
I20260810 23:00:30.510390 19409 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:30.510448 19409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:30.510612 19409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:30.512537 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "c7bd771d35624ec483135a0cac17057d"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "6510f39704f790ae2ad415b38a6494eb"
server_key_iv: "5fc654ccfcab37783449c9c822340a41"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.513026 19409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "c7bd771d35624ec483135a0cac17057d"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "6510f39704f790ae2ad415b38a6494eb"
server_key_iv: "5fc654ccfcab37783449c9c822340a41"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.516649 19409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 23:00:30.518966 19425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.520051 19409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:30.520191 19409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "c7bd771d35624ec483135a0cac17057d"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "6510f39704f790ae2ad415b38a6494eb"
server_key_iv: "5fc654ccfcab37783449c9c822340a41"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.520290 19409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:30.542311 19409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:30.544916 19409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:30.545086 19409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:30.545580 19409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:30.546299 19409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:30.546343 19409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.546377 19409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:30.546424 19409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.555485 19409 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:42107
I20260810 23:00:30.555511 19552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:42107 every 8 connection(s)
I20260810 23:00:30.556260 19409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 23:00:30.564002 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 19409
I20260810 23:00:30.564159 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 23:00:30.564427 30010 external_mini_cluster.cc:1428] Setting key 4f3ad9bd2eddba8400fe3f99a04ebec1
Aug 10 23:00:30 dist-test-slave-6nmv krb5kdc[18986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402830, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.131@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:30.569514 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.558034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:47259 (local address 127.29.78.190:40561)
0810 23:00:30.558162 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.558166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.558924 (+ 758us) server_negotiation.cc:407] Connection header received
0810 23:00:30.560123 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.560126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.560181 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.560276 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:30.561262 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.561752 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.562845 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.563067 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.563183 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.565929 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:30.565933 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:30.565935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:30.565961 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:30.567448 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.567958 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.567961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.567963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.568052 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.568426 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.568429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.568431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.568630 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:30.568737 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.569189 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.569299 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:00:30.570318 19555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.558378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:40561 (local address 127.29.78.131:47259)
0810 23:00:30.558798 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.558833 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.559922 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.560432 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.560438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.560720 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:30.561128 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.561140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.561908 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.561916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.562723 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.562733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.563219 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.563223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.563381 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.563911 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:30.563933 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:30.565763 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:30.567578 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:30.567584 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:30.567597 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:30.567850 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:30.568159 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:30.568164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:30.568166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:30.568319 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:30.568735 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:30.568741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:30.568979 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.569149 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.569457 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":124,"threads_started":1}
I20260810 23:00:30.571357 19553 heartbeater.cc:347] Connected to a master server at 127.29.78.190:40561
I20260810 23:00:30.571609 19553 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:30.572068 19553 heartbeater.cc:511] Master 127.29.78.190:40561 requested a full tablet report, sending...
I20260810 23:00:30.573025 19034 ts_manager.cc:194] Registered new tserver with Master: c7bd771d35624ec483135a0cac17057d (127.29.78.131:42107)
I20260810 23:00:30.573535 19034 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.29.78.131@KRBTEST.COM'} at 127.29.78.131:47259
I20260810 23:00:30.578639 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:00:30.587296 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.580558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.29.78.190:40561)
0810 23:00:30.580669 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.580671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.580877 (+ 206us) server_negotiation.cc:407] Connection header received
0810 23:00:30.580978 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.580980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.581016 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.581075 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:30.581673 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.582162 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.582869 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.583049 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.583117 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.584510 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:30.584512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:30.584514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:30.584529 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:30.585673 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.586199 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.586201 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.586202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.586279 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:30.586543 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:30.586545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:30.586546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:30.586732 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:30.586870 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.587104 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.587193 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:00:30.589892 19034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:00:30.592566 19034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:00:30.607862 19568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.601652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:43187 (local address 127.0.0.1:52448)
0810 23:00:30.602452 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.602464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.602556 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.603347 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.603349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.603364 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:30.603480 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.603485 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.604783 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.604787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.606023 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.606033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.607325 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.607329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.607432 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.607484 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.607592 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.607707 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":5,"thread_start_us":86,"threads_started":1}
I20260810 23:00:30.608407 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.602081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.29.78.130:43187)
0810 23:00:30.602737 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.602763 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.602787 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:00:30.602851 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.602857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.603039 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.603182 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:30.603592 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.604654 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.606145 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.607021 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.607125 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.607258 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.608166 (+ 908us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.608266 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":163,"threads_started":1}
I20260810 23:00:30.608649 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.600803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32838 (local address 127.29.78.129:38943)
0810 23:00:30.601296 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.601312 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.602062 (+ 750us) server_negotiation.cc:407] Connection header received
0810 23:00:30.602424 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.602430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.602650 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.602817 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:30.603482 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.604902 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.606543 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.607785 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.607895 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.608049 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.608381 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.608488 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":93,"threads_started":1}
I20260810 23:00:30.608521 19566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.600699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38943 (local address 127.0.0.1:32838)
0810 23:00:30.601705 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.601754 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.602180 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.602831 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.602836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.602886 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:30.603068 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.603076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.605029 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.605032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.606291 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.606299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.608112 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.608118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.608205 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.608227 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.608309 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.608369 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":83,"threads_started":1}
I20260810 23:00:30.610188 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.600802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42107 (local address 127.0.0.1:40844)
0810 23:00:30.602147 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.602160 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.602248 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.605966 (+ 3718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.605969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.605989 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:30.606123 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.606131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.607771 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.607775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.608929 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.608940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.609732 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.609735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.609855 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.609880 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.609998 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.610066 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1289,"thread_start_us":89,"threads_started":1}
I20260810 23:00:30.610255 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.603176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.29.78.131:42107)
0810 23:00:30.605497 (+ 2321us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.605512 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.605536 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:00:30.605600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.605604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.605764 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.606297 (+ 533us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:30.606464 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.607613 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.609082 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.609609 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.609730 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.609869 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.610039 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.610126 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2183,"thread_start_us":104,"threads_started":1}
I20260810 23:00:30.611565 19328 tablet_service.cc:1467] Processing CreateTablet for tablet d423171832e6498db8c3985d2d2edcae (DEFAULT_TABLE table=test-table [id=5c59d82b24db4f67a93d7d39e76ca7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:30.611840 19173 tablet_service.cc:1467] Processing CreateTablet for tablet d423171832e6498db8c3985d2d2edcae (DEFAULT_TABLE table=test-table [id=5c59d82b24db4f67a93d7d39e76ca7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:30.612679 19173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d423171832e6498db8c3985d2d2edcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:30.612679 19328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d423171832e6498db8c3985d2d2edcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:30.613765 19479 tablet_service.cc:1467] Processing CreateTablet for tablet d423171832e6498db8c3985d2d2edcae (DEFAULT_TABLE table=test-table [id=5c59d82b24db4f67a93d7d39e76ca7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:30.614648 19479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d423171832e6498db8c3985d2d2edcae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:30.619756 19575 tablet_bootstrap.cc:492] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: Bootstrap starting.
I20260810 23:00:30.620182 19574 tablet_bootstrap.cc:492] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Bootstrap starting.
I20260810 23:00:30.620743 19576 tablet_bootstrap.cc:492] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Bootstrap starting.
I20260810 23:00:30.622637 19575 tablet_bootstrap.cc:654] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:30.622659 19576 tablet_bootstrap.cc:654] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:30.623198 19574 tablet_bootstrap.cc:654] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:30.623915 19575 log.cc:824] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:30.623922 19576 log.cc:824] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:30.624402 19574 log.cc:824] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:30.625759 19576 tablet_bootstrap.cc:492] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: No bootstrap required, opened a new log
I20260810 23:00:30.625988 19576 ts_tablet_manager.cc:1397] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:00:30.626188 19574 tablet_bootstrap.cc:492] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: No bootstrap required, opened a new log
I20260810 23:00:30.626188 19575 tablet_bootstrap.cc:492] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: No bootstrap required, opened a new log
I20260810 23:00:30.626356 19575 ts_tablet_manager.cc:1397] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 23:00:30.626356 19574 ts_tablet_manager.cc:1397] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:00:30.629411 19576 raft_consensus.cc:348] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.629637 19576 raft_consensus.cc:374] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:30.629698 19576 raft_consensus.cc:729] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7bd771d35624ec483135a0cac17057d, State: Initialized, Role: FOLLOWER
I20260810 23:00:30.630141 19576 consensus_queue.cc:260] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.630028 19575 raft_consensus.cc:348] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.630028 19574 raft_consensus.cc:348] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.630276 19575 raft_consensus.cc:374] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:30.630276 19574 raft_consensus.cc:374] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:30.630334 19575 raft_consensus.cc:729] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ea2973098f74ad1bd3792e990363d08, State: Initialized, Role: FOLLOWER
I20260810 23:00:30.630333 19574 raft_consensus.cc:729] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ddfeead292a4658bfc80b90be34f325, State: Initialized, Role: FOLLOWER
I20260810 23:00:30.630736 19575 consensus_queue.cc:260] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.630736 19574 consensus_queue.cc:260] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.631080 19576 ts_tablet_manager.cc:1428] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:00:30.631433 19553 heartbeater.cc:503] Master 127.29.78.190:40561 was elected leader, sending a full tablet report...
I20260810 23:00:30.631433 19250 heartbeater.cc:503] Master 127.29.78.190:40561 was elected leader, sending a full tablet report...
I20260810 23:00:30.631664 19575 ts_tablet_manager.cc:1428] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 23:00:30.632207 19404 heartbeater.cc:503] Master 127.29.78.190:40561 was elected leader, sending a full tablet report...
I20260810 23:00:30.632615 19574 ts_tablet_manager.cc:1428] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260810 23:00:30.639521 19251 tablet.cc:2369] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:30.649081 19581 raft_consensus.cc:482] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:30.649209 19581 raft_consensus.cc:504] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.650110 19581 leader_election.cc:288] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7bd771d35624ec483135a0cac17057d (127.29.78.131:42107), 7ddfeead292a4658bfc80b90be34f325 (127.29.78.130:43187)
I20260810 23:00:30.655256 19572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.650427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:60909 (local address 127.29.78.130:43187)
0810 23:00:30.650577 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.650584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.650874 (+ 290us) server_negotiation.cc:407] Connection header received
0810 23:00:30.650957 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.650969 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.651016 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.651090 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:30.651356 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.652124 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.653934 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.654520 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.654640 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.654690 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.655069 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.655135 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:00:30.655256 19573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.650406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:43731 (local address 127.29.78.131:42107)
0810 23:00:30.650547 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.650551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.650686 (+ 135us) server_negotiation.cc:407] Connection header received
0810 23:00:30.650801 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.650803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.650866 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.650963 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:30.651319 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.652124 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.653934 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.654523 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.654640 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.654690 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.655069 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.655135 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:00:30.655336 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.650294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:42107 (local address 127.29.78.129:43731)
0810 23:00:30.650571 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.650588 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.650700 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.651002 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.651005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.651017 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:30.651187 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.651194 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.652240 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.652244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.653784 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.653795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.654650 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.654654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.654873 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.654958 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.655034 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.655189 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":316}
I20260810 23:00:30.655344 19583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.650331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:43187 (local address 127.29.78.129:60909)
0810 23:00:30.650764 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.650778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:30.650879 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:30.651109 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:30.651113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:30.651128 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:30.651250 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.651258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.652240 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.652244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.653784 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:30.653795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.654650 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.654654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:30.654851 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.654953 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 23:00:30.655034 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:00:30.655185 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":305,"thread_start_us":127,"threads_started":1}
I20260810 23:00:30.655946 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d423171832e6498db8c3985d2d2edcae" candidate_uuid: "7ea2973098f74ad1bd3792e990363d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd771d35624ec483135a0cac17057d" is_pre_election: true
I20260810 23:00:30.655977 19351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d423171832e6498db8c3985d2d2edcae" candidate_uuid: "7ea2973098f74ad1bd3792e990363d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ddfeead292a4658bfc80b90be34f325" is_pre_election: true
I20260810 23:00:30.656219 19502 raft_consensus.cc:2454] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ea2973098f74ad1bd3792e990363d08 in term 0.
I20260810 23:00:30.656219 19351 raft_consensus.cc:2454] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ea2973098f74ad1bd3792e990363d08 in term 0.
I20260810 23:00:30.656757 19122 leader_election.cc:302] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7ddfeead292a4658bfc80b90be34f325, 7ea2973098f74ad1bd3792e990363d08, c7bd771d35624ec483135a0cac17057d; no voters:
I20260810 23:00:30.657052 19581 raft_consensus.cc:2781] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:00:30.657150 19581 raft_consensus.cc:482] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:00:30.657232 19581 raft_consensus.cc:3037] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:30.658273 19581 raft_consensus.cc:504] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.658622 19581 leader_election.cc:288] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [CANDIDATE]: Term 1 election: Requested vote from peers c7bd771d35624ec483135a0cac17057d (127.29.78.131:42107), 7ddfeead292a4658bfc80b90be34f325 (127.29.78.130:43187)
I20260810 23:00:30.658952 19351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d423171832e6498db8c3985d2d2edcae" candidate_uuid: "7ea2973098f74ad1bd3792e990363d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ddfeead292a4658bfc80b90be34f325"
I20260810 23:00:30.659054 19351 raft_consensus.cc:3037] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:30.660565 19351 raft_consensus.cc:2454] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ea2973098f74ad1bd3792e990363d08 in term 1.
I20260810 23:00:30.660840 19502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d423171832e6498db8c3985d2d2edcae" candidate_uuid: "7ea2973098f74ad1bd3792e990363d08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7bd771d35624ec483135a0cac17057d"
I20260810 23:00:30.660869 19122 leader_election.cc:302] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ddfeead292a4658bfc80b90be34f325, 7ea2973098f74ad1bd3792e990363d08; no voters:
I20260810 23:00:30.660979 19502 raft_consensus.cc:3037] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:30.661038 19581 raft_consensus.cc:2781] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:30.661238 19581 raft_consensus.cc:686] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [term 1 LEADER]: Becoming Leader. State: Replica: 7ea2973098f74ad1bd3792e990363d08, State: Running, Role: LEADER
I20260810 23:00:30.661557 19581 consensus_queue.cc:237] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } }
I20260810 23:00:30.662286 19502 raft_consensus.cc:2454] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ea2973098f74ad1bd3792e990363d08 in term 1.
I20260810 23:00:30.664815 19033 catalog_manager.cc:5495] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ea2973098f74ad1bd3792e990363d08 (127.29.78.129). New cstate: current_term: 1 leader_uuid: "7ea2973098f74ad1bd3792e990363d08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ea2973098f74ad1bd3792e990363d08" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 38943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 } health_report { overall_health: UNKNOWN } } }
I20260810 23:00:30.683007 19567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.679850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.29.78.129:38943)
0810 23:00:30.679968 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.679972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.679987 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:00:30.680110 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.680112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.680161 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.680242 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:00:30.680448 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.680920 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.681703 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.681998 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.682076 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.682330 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:00:30.682690 (+ 360us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:00:30.682775 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.682832 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.682876 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:00:30.688268 19502 raft_consensus.cc:1261] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 1 FOLLOWER]: Refusing update from remote peer 7ea2973098f74ad1bd3792e990363d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:30.688268 19351 raft_consensus.cc:1261] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 1 FOLLOWER]: Refusing update from remote peer 7ea2973098f74ad1bd3792e990363d08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:30.688928 19581 consensus_queue.cc:1035] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ddfeead292a4658bfc80b90be34f325" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:30.689145 19584 consensus_queue.cc:1035] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7bd771d35624ec483135a0cac17057d" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:30.706633 19033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:30.706799 19033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:30.709113 19033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e38b1cb70bed4e5c92399ff8bc12d2be: Sending DeleteTablet for 3 replicas of tablet d423171832e6498db8c3985d2d2edcae
I20260810 23:00:30.709775 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet d423171832e6498db8c3985d2d2edcae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:30 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:32838
I20260810 23:00:30.709795 19479 tablet_service.cc:1514] Processing DeleteTablet for tablet d423171832e6498db8c3985d2d2edcae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:30 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:40844
I20260810 23:00:30.709909 19328 tablet_service.cc:1514] Processing DeleteTablet for tablet d423171832e6498db8c3985d2d2edcae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:30 UTC) from {username='kudu', principal='kudu/127.29.78.190@KRBTEST.COM'} at 127.0.0.1:52448
I20260810 23:00:30.710325 19600 tablet_replica.cc:330] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: stopping tablet replica
I20260810 23:00:30.710491 19599 tablet_replica.cc:330] T d423171832e6498db8c3985d2d2edcae P 7ea2973098f74ad1bd3792e990363d08: stopping tablet replica
I20260810 23:00:30.710578 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 19103
I20260810 23:00:30.710642 19600 raft_consensus.cc:2229] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:30.716149 19600 raft_consensus.cc:2258] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:30.716562 19603 tablet_replica.cc:330] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: stopping tablet replica
I20260810 23:00:30.716892 19603 raft_consensus.cc:2229] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:30.717185 19603 raft_consensus.cc:2258] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:30.717720 19600 ts_tablet_manager.cc:1905] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:30.718714 19603 ts_tablet_manager.cc:1905] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:30.719907 19600 ts_tablet_manager.cc:1918] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:30.720019 19600 log.cc:1196] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/d423171832e6498db8c3985d2d2edcae
I20260810 23:00:30.720296 19600 ts_tablet_manager.cc:1939] T d423171832e6498db8c3985d2d2edcae P c7bd771d35624ec483135a0cac17057d: Deleting consensus metadata
I20260810 23:00:30.721364 19018 catalog_manager.cc:4841] TS c7bd771d35624ec483135a0cac17057d (127.29.78.131:42107): tablet d423171832e6498db8c3985d2d2edcae (table test-table [id=5c59d82b24db4f67a93d7d39e76ca7e8]) successfully deleted
I20260810 23:00:30.721962 19603 ts_tablet_manager.cc:1918] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:30.722072 19603 log.cc:1196] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/d423171832e6498db8c3985d2d2edcae
I20260810 23:00:30.722474 19603 ts_tablet_manager.cc:1939] T d423171832e6498db8c3985d2d2edcae P 7ddfeead292a4658bfc80b90be34f325: Deleting consensus metadata
W20260810 23:00:30.722611 19017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:38943 (error 108)
I20260810 23:00:30.722896 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 19257
I20260810 23:00:30.725703 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.725405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:38943 (local address 127.0.0.1:32860)
0810 23:00:30.725563 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.725621 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 23:00:30.725910 19017 catalog_manager.cc:4584] TS 7ea2973098f74ad1bd3792e990363d08 (127.29.78.129:38943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d423171832e6498db8c3985d2d2edcae: Network error: Client connection negotiation failed: client connection to 127.29.78.129:38943: connect: Connection refused (error 111)
I20260810 23:00:30.730495 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 19409
I20260810 23:00:30.731434 19569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.730678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:43187 (local address 127.0.0.1:52458)
0810 23:00:30.731301 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:30.731358 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.130:43187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538}
I20260810 23:00:30.737701 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 18999
2026-08-10T23:00:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2239 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T23:00:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:30Z Disabled control of system clock
I20260810 23:00:30.769691 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39683
--webserver_interface=127.29.78.190
--webserver_port=0
--builtin_ntp_servers=127.29.78.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:00:30.878495 19611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:30.878723 19611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:30.878772 19611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:30.882529 19611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:30.882583 19611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:30.882597 19611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:30.882630 19611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:30.882645 19611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:30.889182 19611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:30.890632 19611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:30.892270 19611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:30.900990 19618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:30.901050 19611 server_base.cc:1034] running on GCE node
W20260810 23:00:30.901000 19616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:30.901000 19620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:30.901803 19611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:30.902782 19611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:30.903936 19611 hybrid_clock.cc:648] HybridClock initialized: now 1786402830903918 us; error 29 us; skew 500 ppm
I20260810 23:00:30.905593 19611 webserver.cc:460] Webserver started at http://127.29.78.190:46343/ using document root <none> and password file <none>
I20260810 23:00:30.906183 19611 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:30.906261 19611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:30.906489 19611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:30.908485 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "b60f69e461af49e69fb2333de09c7c53"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "216b75836a99f8abefc033170ba66529"
server_key_iv: "b91d2bf353bc282f8b14d9a1dd244200"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.909040 19611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "b60f69e461af49e69fb2333de09c7c53"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "216b75836a99f8abefc033170ba66529"
server_key_iv: "b91d2bf353bc282f8b14d9a1dd244200"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.912457 19611 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 23:00:30.914803 19627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:30.915990 19611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:30.916127 19611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "b60f69e461af49e69fb2333de09c7c53"
format_stamp: "Formatted at 2026-08-10 23:00:30 on dist-test-slave-6nmv"
server_key: "216b75836a99f8abefc033170ba66529"
server_key_iv: "b91d2bf353bc282f8b14d9a1dd244200"
server_key_version: "encryptionkey@0"
I20260810 23:00:30.916232 19611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:30.926880 19611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:30.927460 19611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:30.927644 19611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:30.933997 19686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:39683 every 8 connection(s)
I20260810 23:00:30.933995 19611 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:39683
I20260810 23:00:30.934813 19611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:30.937582 19687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:30.942456 19687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53: Bootstrap starting.
I20260810 23:00:30.944542 19687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:30.944963 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 19611
I20260810 23:00:30.945103 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:30.945295 19687 log.cc:824] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:30.945374 30010 external_mini_cluster.cc:1428] Setting key 0b415fa940b3d281c5ea193d218c4f03
I20260810 23:00:30.947381 19687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53: No bootstrap required, opened a new log
I20260810 23:00:30.951681 19687 raft_consensus.cc:348] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60f69e461af49e69fb2333de09c7c53" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39683 } }
I20260810 23:00:30.951910 19687 raft_consensus.cc:374] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:30.951972 19687 raft_consensus.cc:729] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60f69e461af49e69fb2333de09c7c53, State: Initialized, Role: FOLLOWER
I20260810 23:00:30.952425 19687 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60f69e461af49e69fb2333de09c7c53" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39683 } }
I20260810 23:00:30.952579 19687 raft_consensus.cc:388] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:30.952679 19687 raft_consensus.cc:482] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:30.952785 19687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:30.953687 19687 raft_consensus.cc:504] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60f69e461af49e69fb2333de09c7c53" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39683 } }
I20260810 23:00:30.954066 19687 leader_election.cc:302] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b60f69e461af49e69fb2333de09c7c53; no voters:
I20260810 23:00:30.954387 19687 leader_election.cc:288] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:30.954479 19692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:30.954690 19692 raft_consensus.cc:686] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [term 1 LEADER]: Becoming Leader. State: Replica: b60f69e461af49e69fb2333de09c7c53, State: Running, Role: LEADER
I20260810 23:00:30.955336 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:30.946416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.29.78.190:39683)
0810 23:00:30.947082 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:30.947103 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:30.947142 (+ 39us) server_negotiation.cc:407] Connection header received
0810 23:00:30.947957 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:30.947977 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:30.948362 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:30.948690 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:30.949116 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.950699 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.951557 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:30.952231 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:30.952408 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:30.953267 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:30.953271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:30.953281 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:30.953614 (+ 333us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:30.953681 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:30.953699 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:30.953789 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:30.954019 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:00:30.954317 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":152,"threads_started":1}
I20260810 23:00:30.955371 19692 consensus_queue.cc:237] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60f69e461af49e69fb2333de09c7c53" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39683 } }
I20260810 23:00:30.955531 19687 sys_catalog.cc:564] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:30.957562 19694 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60f69e461af49e69fb2333de09c7c53. Latest consensus state: current_term: 1 leader_uuid: "b60f69e461af49e69fb2333de09c7c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60f69e461af49e69fb2333de09c7c53" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39683 } } }
I20260810 23:00:30.957715 19694 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:30.957975 19693 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b60f69e461af49e69fb2333de09c7c53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60f69e461af49e69fb2333de09c7c53" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39683 } } }
I20260810 23:00:30.958281 19693 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:30.958138 19708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:30.960556 19708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:30.965639 19708 catalog_manager.cc:1347] Generated new cluster ID: 935778334fff481c9d3bdb52fd5e274b
I20260810 23:00:30.965703 19708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:30.972513 19708 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:30.973376 19708 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:30.986312 19708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53: Generated new TSK 0
I20260810 23:00:30.986961 19708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:30.991111 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.129:0
--local_ip_for_outbound_sockets=127.29.78.129
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39683
--builtin_ntp_servers=127.29.78.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:00:31.114318 19712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:31.114591 19712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:31.114660 19712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:31.118613 19712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:31.118690 19712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:31.118783 19712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.129
I20260810 23:00:31.123673 19712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.29.78.129
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.129
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:31.124941 19712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:31.126561 19712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:31.138230 19723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:31.138190 19720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:31.138599 19721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:31.138938 19712 server_base.cc:1034] running on GCE node
I20260810 23:00:31.139555 19712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:31.140352 19712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:31.143800 19712 hybrid_clock.cc:648] HybridClock initialized: now 1786402831143723 us; error 77 us; skew 500 ppm
I20260810 23:00:31.147277 19712 webserver.cc:460] Webserver started at http://127.29.78.129:41261/ using document root <none> and password file <none>
I20260810 23:00:31.148146 19712 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:31.148291 19712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:31.148578 19712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:31.151083 19712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data/instance:
uuid: "ba5f489f5a564a158f42a37cb65a54ac"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "8e70791b4301803b83849e7c5ab33954"
server_key_iv: "9fe22347e1100a77d09dc10983ad10d6"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.151784 19712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance:
uuid: "ba5f489f5a564a158f42a37cb65a54ac"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "8e70791b4301803b83849e7c5ab33954"
server_key_iv: "9fe22347e1100a77d09dc10983ad10d6"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.156389 19712 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 23:00:31.159969 19730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.162348 19712 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 23:00:31.162552 19712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/wal
uuid: "ba5f489f5a564a158f42a37cb65a54ac"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "8e70791b4301803b83849e7c5ab33954"
server_key_iv: "9fe22347e1100a77d09dc10983ad10d6"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.162706 19712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:31.178774 19712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:31.179438 19712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:31.179641 19712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:31.180303 19712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:31.181255 19712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:31.181330 19712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.181399 19712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:31.181453 19712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.190599 19712 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.129:40607
I20260810 23:00:31.190629 19860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.129:40607 every 8 connection(s)
I20260810 23:00:31.191459 19712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/data/info.pb
I20260810 23:00:31.199520 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 19712
I20260810 23:00:31.199653 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-0/wal/instance
I20260810 23:00:31.199941 30010 external_mini_cluster.cc:1428] Setting key a45a5331692baa11a9aeb4567099137e
I20260810 23:00:31.200223 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.193127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.129:41713 (local address 127.29.78.190:39683)
0810 23:00:31.193256 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.193259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.194278 (+ 1019us) server_negotiation.cc:407] Connection header received
0810 23:00:31.195645 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.195648 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.195696 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.195767 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:31.196628 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.197199 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.198367 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.198575 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.198650 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.199713 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:31.199717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:31.199720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:31.199734 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:31.199762 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:31.199770 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:31.199851 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.200015 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.200079 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:00:31.201128 19863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.193488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39683 (local address 127.29.78.129:41713)
0810 23:00:31.194150 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.194197 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.195458 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.195928 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.195934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.196013 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:31.196477 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.196489 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.197306 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.197308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.198265 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.198274 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.198697 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.198700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.198825 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.199431 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:31.199433 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:31.199471 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:31.199529 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:31.199854 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:31.199867 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:31.199873 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.199999 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.200265 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
I20260810 23:00:31.202143 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.130:0
--local_ip_for_outbound_sockets=127.29.78.130
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39683
--builtin_ntp_servers=127.29.78.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:00:31.202414 19861 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39683
I20260810 23:00:31.202634 19861 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:31.203250 19861 heartbeater.cc:511] Master 127.29.78.190:39683 requested a full tablet report, sending...
I20260810 23:00:31.205025 19644 ts_manager.cc:194] Registered new tserver with Master: ba5f489f5a564a158f42a37cb65a54ac (127.29.78.129:40607)
I20260810 23:00:31.206722 19644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.129:41713
W20260810 23:00:31.308858 19864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:31.309070 19864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:31.309103 19864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:31.312803 19864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:31.312847 19864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:31.312896 19864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.130
I20260810 23:00:31.317382 19864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.29.78.130
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.130
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:31.318342 19864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:31.319599 19864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:31.328225 19870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:31.328241 19872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:31.328258 19869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:31.328770 19864 server_base.cc:1034] running on GCE node
I20260810 23:00:31.329150 19864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:31.329751 19864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:31.330956 19864 hybrid_clock.cc:648] HybridClock initialized: now 1786402831330928 us; error 43 us; skew 500 ppm
I20260810 23:00:31.332733 19864 webserver.cc:460] Webserver started at http://127.29.78.130:32771/ using document root <none> and password file <none>
I20260810 23:00:31.333323 19864 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:31.333413 19864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:31.333645 19864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:31.335625 19864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data/instance:
uuid: "746a3c24e0d044b39d89578d4c06fe07"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "c166ed5a966c85b6f6e56b953b7a2fd3"
server_key_iv: "eb80e38429695caefa3e0d3425e9a50f"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.336177 19864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance:
uuid: "746a3c24e0d044b39d89578d4c06fe07"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "c166ed5a966c85b6f6e56b953b7a2fd3"
server_key_iv: "eb80e38429695caefa3e0d3425e9a50f"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.339707 19864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:00:31.341975 19880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.343091 19864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:31.343218 19864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal
uuid: "746a3c24e0d044b39d89578d4c06fe07"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "c166ed5a966c85b6f6e56b953b7a2fd3"
server_key_iv: "eb80e38429695caefa3e0d3425e9a50f"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.343319 19864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:31.355646 19864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:31.356269 19864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:31.356446 19864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:31.357004 19864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:31.357983 19864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:31.358047 19864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.358104 19864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:31.358152 19864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.367249 19864 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.130:40057
I20260810 23:00:31.367303 20005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.130:40057 every 8 connection(s)
I20260810 23:00:31.368188 19864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/data/info.pb
I20260810 23:00:31.369386 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 19864
I20260810 23:00:31.369522 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal/instance
I20260810 23:00:31.369895 30010 external_mini_cluster.cc:1428] Setting key eb4cc770bc46af9cdccf41bf115005f9
I20260810 23:00:31.372231 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.78.131:0
--local_ip_for_outbound_sockets=127.29.78.131
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39683
--builtin_ntp_servers=127.29.78.148:34275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:00:31.379088 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.370220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:39385 (local address 127.29.78.190:39683)
0810 23:00:31.370414 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.370418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.371603 (+ 1185us) server_negotiation.cc:407] Connection header received
0810 23:00:31.373648 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.373650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.373712 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.373775 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:31.374852 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.375531 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.376686 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.377127 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.377273 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.378447 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:31.378452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:31.378458 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:31.378476 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:31.378507 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:31.378521 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:31.378615 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.378780 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.378866 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 23:00:31.379771 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.370645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39683 (local address 127.29.78.130:39385)
0810 23:00:31.371422 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.371496 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.373402 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.373994 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.374005 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.374114 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:31.374687 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.374702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.375677 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.375679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.376525 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.376535 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.377297 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.377300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.377439 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.378174 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:31.378175 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:31.378213 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:31.378273 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:31.378611 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:31.378617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:31.378620 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.378739 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.378945 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":143,"threads_started":1}
I20260810 23:00:31.380963 20006 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39683
I20260810 23:00:31.381212 20006 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:31.381886 20006 heartbeater.cc:511] Master 127.29.78.190:39683 requested a full tablet report, sending...
I20260810 23:00:31.383018 19644 ts_manager.cc:194] Registered new tserver with Master: 746a3c24e0d044b39d89578d4c06fe07 (127.29.78.130:40057)
I20260810 23:00:31.383561 19644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.130:39385
W20260810 23:00:31.484225 20009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:31.484515 20009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:31.484555 20009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:31.488541 20009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:31.488595 20009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:31.488711 20009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.78.131
I20260810 23:00:31.493288 20009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.78.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.29.78.131
--webserver_port=0
--tserver_master_addrs=127.29.78.190:39683
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.78.131
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:31.494408 20009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:31.495806 20009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:31.505273 20014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:31.505308 20015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:31.505264 20009 server_base.cc:1034] running on GCE node
W20260810 23:00:31.505295 20017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:31.505975 20009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:31.506589 20009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:31.507771 20009 hybrid_clock.cc:648] HybridClock initialized: now 1786402831507759 us; error 39 us; skew 500 ppm
I20260810 23:00:31.509485 20009 webserver.cc:460] Webserver started at http://127.29.78.131:42067/ using document root <none> and password file <none>
I20260810 23:00:31.510061 20009 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:31.510144 20009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:31.510373 20009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:31.512269 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data/instance:
uuid: "f93a64f00dc64173896a85114ab691f9"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "b4a833b476cb5a604c05e7b42197cb9e"
server_key_iv: "c05f4ed175523ec749655b52b0c72959"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.512827 20009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance:
uuid: "f93a64f00dc64173896a85114ab691f9"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "b4a833b476cb5a604c05e7b42197cb9e"
server_key_iv: "c05f4ed175523ec749655b52b0c72959"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.516227 20009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:00:31.518630 20023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.519788 20009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:31.519946 20009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal
uuid: "f93a64f00dc64173896a85114ab691f9"
format_stamp: "Formatted at 2026-08-10 23:00:31 on dist-test-slave-6nmv"
server_key: "b4a833b476cb5a604c05e7b42197cb9e"
server_key_iv: "c05f4ed175523ec749655b52b0c72959"
server_key_version: "encryptionkey@0"
I20260810 23:00:31.520063 20009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:31.529212 20009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:31.529732 20009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:31.530062 20009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:31.530803 20009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:00:31.531756 20009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:00:31.531802 20009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.531838 20009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:00:31.531853 20009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:31.539635 20009 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.131:35723
I20260810 23:00:31.539660 20155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.131:35723 every 8 connection(s)
I20260810 23:00:31.540521 20009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/data/info.pb
I20260810 23:00:31.549057 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.542220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.131:49879 (local address 127.29.78.190:39683)
0810 23:00:31.542346 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.542349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.543300 (+ 951us) server_negotiation.cc:407] Connection header received
0810 23:00:31.544631 (+ 1331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.544634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.544689 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.544784 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:31.545522 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.546060 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.547235 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.547462 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.547569 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.548631 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:31.548639 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:31.548640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:31.548650 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:31.548668 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:31.548674 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:31.548744 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.548875 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.548928 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:00:31.549223 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20009
I20260810 23:00:31.549336 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal/instance
I20260810 23:00:31.549611 30010 external_mini_cluster.cc:1428] Setting key 9e82199e5ce1704a662fcd9e0bbde1b4
I20260810 23:00:31.550081 20158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.542578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39683 (local address 127.29.78.131:49879)
0810 23:00:31.543169 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.543204 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.544429 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.544934 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.544940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.545012 (+ 72us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:31.545394 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.545404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.546180 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.546183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.547104 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.547113 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.547620 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.547624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.547827 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.548378 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:00:31.548379 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:31.548440 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:00:31.548491 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:31.548745 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:31.548756 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:31.548762 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.548881 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.549141 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":121,"threads_started":1}
I20260810 23:00:31.551187 20156 heartbeater.cc:347] Connected to a master server at 127.29.78.190:39683
I20260810 23:00:31.551409 20156 heartbeater.cc:464] Registering TS with master...
I20260810 23:00:31.551944 20156 heartbeater.cc:511] Master 127.29.78.190:39683 requested a full tablet report, sending...
I20260810 23:00:31.552939 19644 ts_manager.cc:194] Registered new tserver with Master: f93a64f00dc64173896a85114ab691f9 (127.29.78.131:35723)
I20260810 23:00:31.553469 19644 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.78.131:49879
I20260810 23:00:31.563136 30010 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:00:31.568717 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.565008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.29.78.190:39683)
0810 23:00:31.565127 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.565131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.565321 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:00:31.565453 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.565456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.565492 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.565569 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:31.565882 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.566369 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.567208 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.567464 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.567556 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.568292 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:31.568295 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:31.568297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:31.568308 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:31.568324 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:31.568334 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:31.568435 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.568555 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.568602 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:00:31.571064 19644 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:00:31.573930 19644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:00:31.588829 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.582098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:35723 (local address 127.0.0.1:55458)
0810 23:00:31.582652 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.582698 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.582812 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.583397 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.583404 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.583439 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:31.583558 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.583564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.585986 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.585988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.587546 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.587558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.588324 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.588327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.588435 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.588478 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.588584 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.588690 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":186,"thread_start_us":83,"threads_started":1}
I20260810 23:00:31.588835 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.582188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.29.78.131:35723)
0810 23:00:31.582532 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.582542 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.582945 (+ 403us) server_negotiation.cc:407] Connection header received
0810 23:00:31.583036 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.583043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.583235 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.583386 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:31.583704 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.584601 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.587697 (+ 3096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.588202 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.588323 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.588499 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.588611 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.588700 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":142,"threads_started":1}
I20260810 23:00:31.589242 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.582180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.130:40057 (local address 127.0.0.1:34160)
0810 23:00:31.582892 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.582906 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.583007 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.584506 (+ 1499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.584509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.584526 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:31.584607 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.584612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.585960 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.585962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.587451 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.587461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.588664 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.588667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.588906 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.588934 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.589019 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.589125 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":247,"thread_start_us":84,"threads_started":1}
I20260810 23:00:31.589344 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.583158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.29.78.129:40607)
0810 23:00:31.583506 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.583518 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.583759 (+ 241us) server_negotiation.cc:407] Connection header received
0810 23:00:31.583893 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.583897 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.584076 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.584469 (+ 393us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:31.584591 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.585761 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.587655 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.588335 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.588438 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.588599 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.589117 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.589202 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":148,"threads_started":1}
I20260810 23:00:31.589435 20171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.582766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40607 (local address 127.0.0.1:55178)
0810 23:00:31.583656 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.583668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.583789 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.584226 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.584229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.584245 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:31.584333 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.584339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.585909 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.585915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.587546 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.587558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.588664 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.588667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.588910 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.588937 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.589019 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.589328 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":486,"thread_start_us":90,"threads_started":1}
I20260810 23:00:31.589840 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.582834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.29.78.130:40057)
0810 23:00:31.584034 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.584044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.584065 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:00:31.584207 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.584211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.584366 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.584507 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:31.584695 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.585844 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.587577 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.588334 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.588463 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.588602 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.589580 (+ 978us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.589673 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1102,"thread_start_us":151,"threads_started":1}
I20260810 23:00:31.592051 19935 tablet_service.cc:1467] Processing CreateTablet for tablet cf81a3b5430c4ef98aad08c2d833cfa9 (DEFAULT_TABLE table=test-table [id=e14d8c6aee3a4a7bb6ac8e5e1898b3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:31.592391 19785 tablet_service.cc:1467] Processing CreateTablet for tablet cf81a3b5430c4ef98aad08c2d833cfa9 (DEFAULT_TABLE table=test-table [id=e14d8c6aee3a4a7bb6ac8e5e1898b3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:31.593243 19935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf81a3b5430c4ef98aad08c2d833cfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:31.593436 19785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf81a3b5430c4ef98aad08c2d833cfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:31.593227 20080 tablet_service.cc:1467] Processing CreateTablet for tablet cf81a3b5430c4ef98aad08c2d833cfa9 (DEFAULT_TABLE table=test-table [id=e14d8c6aee3a4a7bb6ac8e5e1898b3e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:00:31.594175 20080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf81a3b5430c4ef98aad08c2d833cfa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:31.599118 20174 tablet_bootstrap.cc:492] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Bootstrap starting.
I20260810 23:00:31.600519 20176 tablet_bootstrap.cc:492] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Bootstrap starting.
I20260810 23:00:31.601117 20175 tablet_bootstrap.cc:492] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: Bootstrap starting.
I20260810 23:00:31.601491 20174 tablet_bootstrap.cc:654] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:31.602350 20174 log.cc:824] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:31.602610 20176 tablet_bootstrap.cc:654] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:31.602919 20175 tablet_bootstrap.cc:654] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:31.603394 20176 log.cc:824] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:31.603724 20175 log.cc:824] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:31.604473 20174 tablet_bootstrap.cc:492] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: No bootstrap required, opened a new log
I20260810 23:00:31.604650 20174 ts_tablet_manager.cc:1397] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 23:00:31.605178 20176 tablet_bootstrap.cc:492] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: No bootstrap required, opened a new log
I20260810 23:00:31.605351 20176 ts_tablet_manager.cc:1397] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 23:00:31.605355 20175 tablet_bootstrap.cc:492] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: No bootstrap required, opened a new log
I20260810 23:00:31.605548 20175 ts_tablet_manager.cc:1397] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 23:00:31.608675 20175 raft_consensus.cc:348] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.608917 20175 raft_consensus.cc:374] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:31.608795 20174 raft_consensus.cc:348] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.608973 20175 raft_consensus.cc:729] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5f489f5a564a158f42a37cb65a54ac, State: Initialized, Role: FOLLOWER
I20260810 23:00:31.609012 20174 raft_consensus.cc:374] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:31.609076 20174 raft_consensus.cc:729] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 746a3c24e0d044b39d89578d4c06fe07, State: Initialized, Role: FOLLOWER
I20260810 23:00:31.609396 20175 consensus_queue.cc:260] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.609308 20176 raft_consensus.cc:348] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.609534 20176 raft_consensus.cc:374] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:31.609470 20174 consensus_queue.cc:260] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.609594 20176 raft_consensus.cc:729] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93a64f00dc64173896a85114ab691f9, State: Initialized, Role: FOLLOWER
I20260810 23:00:31.610033 20176 consensus_queue.cc:260] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.610124 20006 heartbeater.cc:503] Master 127.29.78.190:39683 was elected leader, sending a full tablet report...
I20260810 23:00:31.610378 20175 ts_tablet_manager.cc:1428] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260810 23:00:31.610378 20174 ts_tablet_manager.cc:1428] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 23:00:31.610657 19861 heartbeater.cc:503] Master 127.29.78.190:39683 was elected leader, sending a full tablet report...
I20260810 23:00:31.610774 20176 ts_tablet_manager.cc:1428] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 23:00:31.611039 20156 heartbeater.cc:503] Master 127.29.78.190:39683 was elected leader, sending a full tablet report...
W20260810 23:00:31.619053 20007 tablet.cc:2369] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:00:31.692617 19862 tablet.cc:2369] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:00:31.791514 20157 tablet.cc:2369] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:31.819270 20182 raft_consensus.cc:482] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:00:31.819514 20182 raft_consensus.cc:504] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.820747 20182 leader_election.cc:288] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93a64f00dc64173896a85114ab691f9 (127.29.78.131:35723), ba5f489f5a564a158f42a37cb65a54ac (127.29.78.129:40607)
I20260810 23:00:31.825518 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.821066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:44243 (local address 127.29.78.131:35723)
0810 23:00:31.821194 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.821197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.821210 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:00:31.821260 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.821262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.821301 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.821396 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:31.821839 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.822671 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.824199 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.824837 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.824972 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.825016 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.825316 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.825385 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:00:31.825518 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.821456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.29.78.130:47899 (local address 127.29.78.129:40607)
0810 23:00:31.821590 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.821593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.821605 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:00:31.821725 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.821727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.821795 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.821894 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:00:31.822145 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.822856 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.824240 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.824837 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.824963 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.825016 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.825315 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.825385 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:00:31.825532 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.820939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.131:35723 (local address 127.29.78.130:44243)
0810 23:00:31.821050 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.821068 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.821161 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.821528 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.821531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.821553 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:31.821692 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.821700 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.822789 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.822792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.824066 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.824078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.824969 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.824973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.825152 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.825199 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.825280 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.825376 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":57}
I20260810 23:00:31.825541 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.821190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40607 (local address 127.29.78.130:47899)
0810 23:00:31.821537 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.821549 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:31.821623 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:31.821904 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:31.821906 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:31.821930 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:00:31.822026 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.822030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.822968 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.822972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.824129 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:31.824138 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.824984 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.824986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:31.825157 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.825206 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:00:31.825280 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:00:31.825377 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":70,"thread_start_us":152,"threads_started":1}
I20260810 23:00:31.826316 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf81a3b5430c4ef98aad08c2d833cfa9" candidate_uuid: "746a3c24e0d044b39d89578d4c06fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93a64f00dc64173896a85114ab691f9" is_pre_election: true
I20260810 23:00:31.826627 19806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf81a3b5430c4ef98aad08c2d833cfa9" candidate_uuid: "746a3c24e0d044b39d89578d4c06fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5f489f5a564a158f42a37cb65a54ac" is_pre_election: true
I20260810 23:00:31.826666 20101 raft_consensus.cc:2454] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 746a3c24e0d044b39d89578d4c06fe07 in term 0.
I20260810 23:00:31.826886 19806 raft_consensus.cc:2454] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 746a3c24e0d044b39d89578d4c06fe07 in term 0.
I20260810 23:00:31.827229 19883 leader_election.cc:302] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 746a3c24e0d044b39d89578d4c06fe07, f93a64f00dc64173896a85114ab691f9; no voters:
I20260810 23:00:31.827520 20182 raft_consensus.cc:2781] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:00:31.827620 20182 raft_consensus.cc:482] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:00:31.827684 20182 raft_consensus.cc:3037] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:31.828758 20182 raft_consensus.cc:504] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.829098 20182 leader_election.cc:288] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [CANDIDATE]: Term 1 election: Requested vote from peers f93a64f00dc64173896a85114ab691f9 (127.29.78.131:35723), ba5f489f5a564a158f42a37cb65a54ac (127.29.78.129:40607)
I20260810 23:00:31.829481 20101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf81a3b5430c4ef98aad08c2d833cfa9" candidate_uuid: "746a3c24e0d044b39d89578d4c06fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93a64f00dc64173896a85114ab691f9"
I20260810 23:00:31.829526 19806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf81a3b5430c4ef98aad08c2d833cfa9" candidate_uuid: "746a3c24e0d044b39d89578d4c06fe07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba5f489f5a564a158f42a37cb65a54ac"
I20260810 23:00:31.829576 20101 raft_consensus.cc:3037] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:31.829676 19806 raft_consensus.cc:3037] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:31.830672 20101 raft_consensus.cc:2454] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746a3c24e0d044b39d89578d4c06fe07 in term 1.
I20260810 23:00:31.830703 19806 raft_consensus.cc:2454] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 746a3c24e0d044b39d89578d4c06fe07 in term 1.
I20260810 23:00:31.830961 19883 leader_election.cc:302] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 746a3c24e0d044b39d89578d4c06fe07, f93a64f00dc64173896a85114ab691f9; no voters:
I20260810 23:00:31.831142 20182 raft_consensus.cc:2781] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:31.831372 20182 raft_consensus.cc:686] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 1 LEADER]: Becoming Leader. State: Replica: 746a3c24e0d044b39d89578d4c06fe07, State: Running, Role: LEADER
I20260810 23:00:31.831737 20182 consensus_queue.cc:237] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } }
I20260810 23:00:31.834704 19642 catalog_manager.cc:5495] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 746a3c24e0d044b39d89578d4c06fe07 (127.29.78.130). New cstate: current_term: 1 leader_uuid: "746a3c24e0d044b39d89578d4c06fe07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "746a3c24e0d044b39d89578d4c06fe07" member_type: VOTER last_known_addr { host: "127.29.78.130" port: 40057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 } health_report { overall_health: UNKNOWN } } }
I20260810 23:00:31.877384 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.873899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34166 (local address 127.29.78.130:40057)
0810 23:00:31.874039 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:31.874044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:31.874059 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:00:31.874183 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:31.874186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:31.874243 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:31.874354 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:00:31.874602 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.875107 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.875959 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:31.876177 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:31.876289 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:31.876599 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:00:31.876988 (+ 389us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:00:31.877119 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:31.877170 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:00:31.877220 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:00:31.882896 20101 raft_consensus.cc:1261] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 1 FOLLOWER]: Refusing update from remote peer 746a3c24e0d044b39d89578d4c06fe07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:31.882894 19806 raft_consensus.cc:1261] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac [term 1 FOLLOWER]: Refusing update from remote peer 746a3c24e0d044b39d89578d4c06fe07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:31.883590 20185 consensus_queue.cc:1035] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba5f489f5a564a158f42a37cb65a54ac" member_type: VOTER last_known_addr { host: "127.29.78.129" port: 40607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:31.883749 20182 consensus_queue.cc:1035] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93a64f00dc64173896a85114ab691f9" member_type: VOTER last_known_addr { host: "127.29.78.131" port: 35723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:31.891181 20188 mvcc.cc:205] Tried to move back new op lower bound from 7317105999385313280 to 7317105999185072128. Current Snapshot: MvccSnapshot[applied={T|T < 7317105999385313280}]
I20260810 23:00:31.891361 20190 mvcc.cc:205] Tried to move back new op lower bound from 7317105999385313280 to 7317105999185072128. Current Snapshot: MvccSnapshot[applied={T|T < 7317105999385313280}]
I20260810 23:00:31.899513 19642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:31.899669 19642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:00:31.902110 19642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b60f69e461af49e69fb2333de09c7c53: Sending DeleteTablet for 3 replicas of tablet cf81a3b5430c4ef98aad08c2d833cfa9
I20260810 23:00:31.902813 19935 tablet_service.cc:1514] Processing DeleteTablet for tablet cf81a3b5430c4ef98aad08c2d833cfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:31 UTC) from {username='slave'} at 127.0.0.1:34160
I20260810 23:00:31.902864 20080 tablet_service.cc:1514] Processing DeleteTablet for tablet cf81a3b5430c4ef98aad08c2d833cfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:31 UTC) from {username='slave'} at 127.0.0.1:55458
I20260810 23:00:31.902899 19785 tablet_service.cc:1514] Processing DeleteTablet for tablet cf81a3b5430c4ef98aad08c2d833cfa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:00:31 UTC) from {username='slave'} at 127.0.0.1:55178
I20260810 23:00:31.903388 20200 tablet_replica.cc:330] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: stopping tablet replica
I20260810 23:00:31.903520 20202 tablet_replica.cc:330] T cf81a3b5430c4ef98aad08c2d833cfa9 P ba5f489f5a564a158f42a37cb65a54ac: stopping tablet replica
I20260810 23:00:31.903810 20200 raft_consensus.cc:2229] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:00:31.904006 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 19712
I20260810 23:00:31.903999 20201 tablet_replica.cc:330] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: stopping tablet replica
I20260810 23:00:31.904312 20201 raft_consensus.cc:2229] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:00:31.905493 20201 raft_consensus.cc:2258] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:31.908284 20200 raft_consensus.cc:2258] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:00:31.909729 20201 ts_tablet_manager.cc:1905] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:31.912737 20201 ts_tablet_manager.cc:1918] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:31.912828 20201 log.cc:1196] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-1/wal/wals/cf81a3b5430c4ef98aad08c2d833cfa9
I20260810 23:00:31.913130 20201 ts_tablet_manager.cc:1939] T cf81a3b5430c4ef98aad08c2d833cfa9 P 746a3c24e0d044b39d89578d4c06fe07: Deleting consensus metadata
I20260810 23:00:31.914224 19629 catalog_manager.cc:4841] TS 746a3c24e0d044b39d89578d4c06fe07 (127.29.78.130:40057): tablet cf81a3b5430c4ef98aad08c2d833cfa9 (table test-table [id=e14d8c6aee3a4a7bb6ac8e5e1898b3e2]) successfully deleted
I20260810 23:00:31.915141 20200 ts_tablet_manager.cc:1905] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:00:31.917269 20200 ts_tablet_manager.cc:1918] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:00:31.917382 20200 log.cc:1196] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Deleting WAL directory at /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402578782454-30010-0/minicluster-data/ts-2/wal/wals/cf81a3b5430c4ef98aad08c2d833cfa9
I20260810 23:00:31.917666 20200 ts_tablet_manager.cc:1939] T cf81a3b5430c4ef98aad08c2d833cfa9 P f93a64f00dc64173896a85114ab691f9: Deleting consensus metadata
I20260810 23:00:31.918654 19630 catalog_manager.cc:4841] TS f93a64f00dc64173896a85114ab691f9 (127.29.78.131:35723): tablet cf81a3b5430c4ef98aad08c2d833cfa9 (table test-table [id=e14d8c6aee3a4a7bb6ac8e5e1898b3e2]) successfully deleted
W20260810 23:00:31.922072 19630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.78.129:40607 (error 108)
I20260810 23:00:31.922313 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 19864
I20260810 23:00:31.924067 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:31.923710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.129:40607 (local address 127.0.0.1:55188)
0810 23:00:31.923907 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:31.923975 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260810 23:00:31.924357 19630 catalog_manager.cc:4584] TS ba5f489f5a564a158f42a37cb65a54ac (127.29.78.129:40607): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf81a3b5430c4ef98aad08c2d833cfa9: Network error: Client connection negotiation failed: client connection to 127.29.78.129:40607: connect: Connection refused (error 111)
I20260810 23:00:31.933319 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20009
I20260810 23:00:31.940896 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 19611
2026-08-10T23:00:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1201 ms)
[----------] 6 tests from EncryptionPolicyTest (10327 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:32 dist-test-slave-6nmv krb5kdc[20206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:32 dist-test-slave-6nmv krb5kdc[20206](info): set up 2 sockets
Aug 10 23:00:32 dist-test-slave-6nmv krb5kdc[20206](info): commencing operation
krb5kdc: starting...
W20260810 23:00:33.023152 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:33 dist-test-slave-6nmv krb5kdc[20206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:33.140861 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:33.247395 20223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:33.247668 20223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:33.247741 20223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:33.251325 20223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:33.251407 20223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:33.251447 20223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:33.251503 20223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:33.251540 20223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:33.256539 20223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34315
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:33.257586 20223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:33.258576 20223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:00:33.267082 20223 server_base.cc:1034] running on GCE node
W20260810 23:00:33.267019 20231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:33.267019 20229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:33.267027 20228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:33.267830 20223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:33.268867 20223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:33.270049 20223 hybrid_clock.cc:648] HybridClock initialized: now 1786402833270036 us; error 33 us; skew 500 ppm
Aug 10 23:00:33 dist-test-slave-6nmv krb5kdc[20206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:33.272732 20223 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:33.273582 20223 webserver.cc:460] Webserver started at http://127.0.0.1:42371/ using document root <none> and password file <none>
I20260810 23:00:33.274144 20223 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:33.274241 20223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:33.274470 20223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:33.276309 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "cd2a699980dc4a09898d6cde7c46ba97"
format_stamp: "Formatted at 2026-08-10 23:00:33 on dist-test-slave-6nmv"
server_key: "5135f2251de91429bbb142e2bd730ebd"
server_key_iv: "1ed10fe6289c225ab18d9ebb257ec23d"
server_key_version: "encryptionkey@0"
I20260810 23:00:33.276885 20223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "cd2a699980dc4a09898d6cde7c46ba97"
format_stamp: "Formatted at 2026-08-10 23:00:33 on dist-test-slave-6nmv"
server_key: "5135f2251de91429bbb142e2bd730ebd"
server_key_iv: "1ed10fe6289c225ab18d9ebb257ec23d"
server_key_version: "encryptionkey@0"
I20260810 23:00:33.280292 20223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 23:00:33.282636 20238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:33.283811 20223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:00:33.283944 20223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "cd2a699980dc4a09898d6cde7c46ba97"
format_stamp: "Formatted at 2026-08-10 23:00:33 on dist-test-slave-6nmv"
server_key: "5135f2251de91429bbb142e2bd730ebd"
server_key_iv: "1ed10fe6289c225ab18d9ebb257ec23d"
server_key_version: "encryptionkey@0"
I20260810 23:00:33.284078 20223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:33.299995 20223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:33.302932 20223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:33.303136 20223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:33.310331 20223 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34315
I20260810 23:00:33.310362 20296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34315 every 8 connection(s)
I20260810 23:00:33.311220 20223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:33.313906 20297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:33.317085 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20223
I20260810 23:00:33.317232 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:33.317528 30010 external_mini_cluster.cc:1428] Setting key 7b1fd80f37c33e03919b68c897592497
I20260810 23:00:33.319687 20297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97: Bootstrap starting.
I20260810 23:00:33.321697 20297 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:33.322453 20297 log.cc:824] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:33.324237 20297 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97: No bootstrap required, opened a new log
Aug 10 23:00:33 dist-test-slave-6nmv krb5kdc[20206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:00:33.328023 20297 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2a699980dc4a09898d6cde7c46ba97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260810 23:00:33.328212 20297 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:33.328277 20297 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd2a699980dc4a09898d6cde7c46ba97, State: Initialized, Role: FOLLOWER
I20260810 23:00:33.328706 20297 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2a699980dc4a09898d6cde7c46ba97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260810 23:00:33.328821 20297 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:33.328914 20297 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:33.329017 20297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:33.330174 20297 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2a699980dc4a09898d6cde7c46ba97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260810 23:00:33.330519 20297 leader_election.cc:302] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd2a699980dc4a09898d6cde7c46ba97; no voters:
I20260810 23:00:33.330812 20297 leader_election.cc:288] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:33.330907 20302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:33.331138 20302 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [term 1 LEADER]: Becoming Leader. State: Replica: cd2a699980dc4a09898d6cde7c46ba97, State: Running, Role: LEADER
I20260810 23:00:33.331492 20302 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2a699980dc4a09898d6cde7c46ba97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } }
I20260810 23:00:33.331971 20297 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:33.332501 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:33.318938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54519 (local address 127.0.0.1:34315)
0810 23:00:33.319333 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:33.319347 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:33.319380 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:00:33.320037 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:33.320053 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:33.320351 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:33.320655 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:33.321436 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:33.322568 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:33.323631 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:33.324102 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:33.324262 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:33.327037 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:33.327041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:33.327053 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:33.327078 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:33.329137 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:33.329665 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:33.329669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:33.329673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:33.329772 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:33.330153 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:33.330155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:33.330156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:33.330556 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:33.330810 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:33.331185 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 23:00:33.331579 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":126,"threads_started":1}
I20260810 23:00:33.332692 20302 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd2a699980dc4a09898d6cde7c46ba97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2a699980dc4a09898d6cde7c46ba97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260810 23:00:33.332875 20302 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:33.333137 20302 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd2a699980dc4a09898d6cde7c46ba97. Latest consensus state: current_term: 1 leader_uuid: "cd2a699980dc4a09898d6cde7c46ba97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd2a699980dc4a09898d6cde7c46ba97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34315 } } }
I20260810 23:00:33.333278 20302 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:33.333555 20314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:33.335816 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:33.336740 20314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:33.342608 20314 catalog_manager.cc:1347] Generated new cluster ID: 8bbbc9a6c7a94ba8a9e0506838bc674f
I20260810 23:00:33.342669 20314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:33.347268 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:33.338119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58735 (local address 127.0.0.1:34315)
0810 23:00:33.338225 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:33.338229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:33.338712 (+ 483us) server_negotiation.cc:407] Connection header received
0810 23:00:33.338799 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:33.338804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:33.338847 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:33.338922 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:33.339794 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:33.340515 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:33.341447 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:33.341693 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:33.341807 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:33.343521 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:33.343524 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:33.343527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:33.343551 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:33.345249 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:33.345751 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:33.345754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:33.345756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:33.345885 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:33.346223 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:33.346227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:33.346229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:33.346427 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:33.346591 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:33.346849 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 23:00:33.347117 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:00:33.362694 20314 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:33.363793 20314 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:33.373237 20314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd2a699980dc4a09898d6cde7c46ba97: Generated new TSK 0
I20260810 23:00:33.373880 20314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:33.380429 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20223
W20260810 23:00:33.389976 20204 connection.cc:535] client connection to 127.0.0.1:34315 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34315): Connection reset by peer (error 104)
2026-08-10T23:00:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:33 dist-test-slave-6nmv krb5kdc[20332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:33 dist-test-slave-6nmv krb5kdc[20332](info): set up 2 sockets
Aug 10 23:00:33 dist-test-slave-6nmv krb5kdc[20332](info): commencing operation
krb5kdc: starting...
W20260810 23:00:34.417429 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:34 dist-test-slave-6nmv krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402834, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:34.530911 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:34.638275 20346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:34.638551 20346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:34.638621 20346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:34.642431 20346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:34.642555 20346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:34.642599 20346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:34.642637 20346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:34.642673 20346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:34.647817 20346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:34.648870 20346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:34.649843 20346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:34.658115 20355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:34.658203 20346 server_base.cc:1034] running on GCE node
W20260810 23:00:34.658149 20353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:34.658152 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:34.658957 20346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:34.660126 20346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:34.661330 20346 hybrid_clock.cc:648] HybridClock initialized: now 1786402834661315 us; error 34 us; skew 500 ppm
Aug 10 23:00:34 dist-test-slave-6nmv krb5kdc[20332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402834, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:34.664418 20346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:34.665401 20346 webserver.cc:460] Webserver started at http://127.0.0.1:41653/ using document root <none> and password file <none>
I20260810 23:00:34.666127 20346 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:34.666244 20346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:34.666563 20346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:34.668741 20346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "916996734b5a4919b7748964850cee42"
format_stamp: "Formatted at 2026-08-10 23:00:34 on dist-test-slave-6nmv"
server_key: "67d6c1696d7fbacebd15b67d746e694f"
server_key_iv: "05cfe950ba77f7aad5e4e5816ad6e96a"
server_key_version: "encryptionkey@0"
I20260810 23:00:34.669394 20346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "916996734b5a4919b7748964850cee42"
format_stamp: "Formatted at 2026-08-10 23:00:34 on dist-test-slave-6nmv"
server_key: "67d6c1696d7fbacebd15b67d746e694f"
server_key_iv: "05cfe950ba77f7aad5e4e5816ad6e96a"
server_key_version: "encryptionkey@0"
I20260810 23:00:34.673408 20346 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 23:00:34.676038 20362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:34.677244 20346 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:34.677399 20346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "916996734b5a4919b7748964850cee42"
format_stamp: "Formatted at 2026-08-10 23:00:34 on dist-test-slave-6nmv"
server_key: "67d6c1696d7fbacebd15b67d746e694f"
server_key_iv: "05cfe950ba77f7aad5e4e5816ad6e96a"
server_key_version: "encryptionkey@0"
I20260810 23:00:34.677523 20346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:34.698311 20346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:34.701546 20346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:34.701749 20346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:34.709755 20346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36795
I20260810 23:00:34.709784 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36795 every 8 connection(s)
I20260810 23:00:34.710736 20346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:34.713346 20421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:34.716583 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20346
I20260810 23:00:34.716750 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:34.717095 30010 external_mini_cluster.cc:1428] Setting key 4dfceb43475590e4973f9c575e444365
I20260810 23:00:34.719491 20421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42: Bootstrap starting.
I20260810 23:00:34.721622 20421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:34.722395 20421 log.cc:824] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:34.724354 20421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42: No bootstrap required, opened a new log
I20260810 23:00:34.726222 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:34.718524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42831 (local address 127.0.0.1:36795)
0810 23:00:34.718930 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:34.718943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:34.718976 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:00:34.719567 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:34.719608 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:34.719968 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:34.720284 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:34.720685 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:34.721627 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:34.722570 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:34.723217 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:34.723402 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:34.724309 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:34.724314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:34.724325 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:34.724749 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:34.724799 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:34.724821 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:34.724956 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:34.725146 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:00:34.725273 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":122,"threads_started":1}
I20260810 23:00:34.727680 20421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916996734b5a4919b7748964850cee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260810 23:00:34.727907 20421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:34.727977 20421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916996734b5a4919b7748964850cee42, State: Initialized, Role: FOLLOWER
I20260810 23:00:34.728489 20421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916996734b5a4919b7748964850cee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260810 23:00:34.728637 20421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:34.728701 20421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:34.728790 20421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:34.729135 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:34.729880 20421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916996734b5a4919b7748964850cee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260810 23:00:34.730233 20421 leader_election.cc:302] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 916996734b5a4919b7748964850cee42; no voters:
I20260810 23:00:34.730582 20421 leader_election.cc:288] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:34.730675 20428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:34.730932 20428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [term 1 LEADER]: Becoming Leader. State: Replica: 916996734b5a4919b7748964850cee42, State: Running, Role: LEADER
I20260810 23:00:34.731289 20428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916996734b5a4919b7748964850cee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } }
I20260810 23:00:34.731818 20421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:34.732496 20428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 916996734b5a4919b7748964850cee42. Latest consensus state: current_term: 1 leader_uuid: "916996734b5a4919b7748964850cee42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916996734b5a4919b7748964850cee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } } }
I20260810 23:00:34.732648 20428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:34.732921 20431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "916996734b5a4919b7748964850cee42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "916996734b5a4919b7748964850cee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36795 } } }
I20260810 23:00:34.733065 20431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:34.733130 20441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:34.735810 20441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:34.738293 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:34.731442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51539 (local address 127.0.0.1:36795)
0810 23:00:34.731537 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:34.731541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:34.734089 (+ 2548us) server_negotiation.cc:407] Connection header received
0810 23:00:34.734209 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:34.734213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:34.734257 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:34.734331 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:34.734582 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:34.735239 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:34.736538 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:34.736774 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:34.736883 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:34.737764 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:34.737768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:34.737771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:34.737784 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:34.737839 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:34.737855 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:34.737948 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:34.738140 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 23:00:34.738174 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:00:34.741379 20441 catalog_manager.cc:1347] Generated new cluster ID: 88725e4309fa4725b766afe541ce574a
I20260810 23:00:34.741456 20441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:34.769850 20441 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:34.770824 20441 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:34.776952 20441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 916996734b5a4919b7748964850cee42: Generated new TSK 0
I20260810 23:00:34.777586 20441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:34.784766 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20346
W20260810 23:00:34.791760 20330 connection.cc:535] client connection to 127.0.0.1:36795 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36795): Connection reset by peer (error 104)
2026-08-10T23:00:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1402 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:34 dist-test-slave-6nmv krb5kdc[20454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:34 dist-test-slave-6nmv krb5kdc[20454](info): set up 2 sockets
Aug 10 23:00:34 dist-test-slave-6nmv krb5kdc[20454](info): commencing operation
krb5kdc: starting...
W20260810 23:00:35.822875 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:35 dist-test-slave-6nmv krb5kdc[20454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:35.944253 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34089
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:36.049209 20469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:36.049526 20469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:36.049602 20469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:36.053318 20469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:36.053393 20469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:36.053443 20469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:36.053476 20469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:36.053509 20469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:36.058136 20469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:36.059043 20469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:36.059949 20469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:36.068209 20475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:36.068387 20469 server_base.cc:1034] running on GCE node
W20260810 23:00:36.068210 20478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:36.068260 20476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:36.069074 20469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:36.070137 20469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:36.071349 20469 hybrid_clock.cc:648] HybridClock initialized: now 1786402836071333 us; error 31 us; skew 500 ppm
Aug 10 23:00:36 dist-test-slave-6nmv krb5kdc[20454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:36.078054 20469 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:36.078799 20469 webserver.cc:460] Webserver started at http://127.0.0.1:34607/ using document root <none> and password file <none>
I20260810 23:00:36.079277 20469 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:36.079329 20469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:36.079485 20469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:36.081301 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "2a0e1039ab54437183009b580937b2dc"
format_stamp: "Formatted at 2026-08-10 23:00:36 on dist-test-slave-6nmv"
server_key: "e77009034ff5be08fa625c7fb615ae09"
server_key_iv: "d399c54ba3ac3a52d6073f94164c9bd5"
server_key_version: "encryptionkey@0"
I20260810 23:00:36.081768 20469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "2a0e1039ab54437183009b580937b2dc"
format_stamp: "Formatted at 2026-08-10 23:00:36 on dist-test-slave-6nmv"
server_key: "e77009034ff5be08fa625c7fb615ae09"
server_key_iv: "d399c54ba3ac3a52d6073f94164c9bd5"
server_key_version: "encryptionkey@0"
I20260810 23:00:36.085040 20469 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260810 23:00:36.087316 20486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:36.088471 20469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:36.088589 20469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "2a0e1039ab54437183009b580937b2dc"
format_stamp: "Formatted at 2026-08-10 23:00:36 on dist-test-slave-6nmv"
server_key: "e77009034ff5be08fa625c7fb615ae09"
server_key_iv: "d399c54ba3ac3a52d6073f94164c9bd5"
server_key_version: "encryptionkey@0"
I20260810 23:00:36.088680 20469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:36.092880 20469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:36.095884 20469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:36.096047 20469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:36.102773 20469 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34089
I20260810 23:00:36.102818 20546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34089 every 8 connection(s)
I20260810 23:00:36.103621 20469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:36.106173 20547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:36.109691 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20469
I20260810 23:00:36.109889 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:36.110168 30010 external_mini_cluster.cc:1428] Setting key cd5a232965df9422d04876559c3f8423
I20260810 23:00:36.111579 20547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc: Bootstrap starting.
I20260810 23:00:36.113802 20547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:36.114708 20547 log.cc:824] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:36.115222 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:36.111439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51883 (local address 127.0.0.1:34089)
0810 23:00:36.111933 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:36.111946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:36.111980 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:00:36.112565 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:36.112580 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:36.112833 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:36.113176 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:36.113930 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:36.113935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:36.113950 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:36.113966 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:36.114018 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:36.114036 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:36.114149 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:36.114250 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:00:36.114419 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":165,"threads_started":1}
I20260810 23:00:36.116804 20547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc: No bootstrap required, opened a new log
I20260810 23:00:36.118101 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:36.121335 20547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0e1039ab54437183009b580937b2dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34089 } }
I20260810 23:00:36.121599 20547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:36.121694 20547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a0e1039ab54437183009b580937b2dc, State: Initialized, Role: FOLLOWER
I20260810 23:00:36.122187 20550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:36.120480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54943 (local address 127.0.0.1:34089)
0810 23:00:36.120585 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:36.120592 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:36.120800 (+ 208us) server_negotiation.cc:407] Connection header received
0810 23:00:36.120966 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:36.120969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:36.121006 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:36.121092 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:36.121801 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:36.121804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:36.121829 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:36.121843 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:36.121867 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:36.121879 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:36.121978 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:36.122046 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 23:00:36.122081 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 23:00:36.122228 20547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0e1039ab54437183009b580937b2dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34089 } }
I20260810 23:00:36.122393 20547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:36.122467 20547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:36.122572 20547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:36.123603 20547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0e1039ab54437183009b580937b2dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34089 } }
I20260810 23:00:36.123945 20547 leader_election.cc:302] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a0e1039ab54437183009b580937b2dc; no voters:
I20260810 23:00:36.124243 20547 leader_election.cc:288] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:36.124346 20562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:36.124603 20562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [term 1 LEADER]: Becoming Leader. State: Replica: 2a0e1039ab54437183009b580937b2dc, State: Running, Role: LEADER
I20260810 23:00:36.124954 20562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0e1039ab54437183009b580937b2dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34089 } }
I20260810 23:00:36.125443 20547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:36.126602 20564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a0e1039ab54437183009b580937b2dc. Latest consensus state: current_term: 1 leader_uuid: "2a0e1039ab54437183009b580937b2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0e1039ab54437183009b580937b2dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34089 } } }
I20260810 23:00:36.126729 20564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:36.127053 20563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a0e1039ab54437183009b580937b2dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a0e1039ab54437183009b580937b2dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34089 } } }
I20260810 23:00:36.127210 20563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:36.127142 20572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:36.130214 20572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:36.135406 20572 catalog_manager.cc:1347] Generated new cluster ID: ce17cc4324c240e7aeaf0c65f585cb51
I20260810 23:00:36.135484 20572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:36.152808 20572 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:36.153110 20572 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:36.160992 20572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a0e1039ab54437183009b580937b2dc: Generated new TSK 0
I20260810 23:00:36.161620 20572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:36.164944 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20469
2026-08-10T23:00:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:36 dist-test-slave-6nmv krb5kdc[20585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:36 dist-test-slave-6nmv krb5kdc[20585](info): set up 2 sockets
Aug 10 23:00:36 dist-test-slave-6nmv krb5kdc[20585](info): commencing operation
krb5kdc: starting...
W20260810 23:00:37.236022 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:37 dist-test-slave-6nmv krb5kdc[20585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:37.354959 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41037
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:37.459467 20602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:37.459729 20602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:37.459802 20602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:37.464167 20602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:37.464246 20602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:37.464283 20602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:37.464318 20602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:37.464354 20602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:37.469362 20602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:37.470293 20602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:37.471225 20602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:37.479060 20607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:37.479049 20602 server_base.cc:1034] running on GCE node
W20260810 23:00:37.479092 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:37.479036 20608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:37.479804 20602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:37.480753 20602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:37.481884 20602 hybrid_clock.cc:648] HybridClock initialized: now 1786402837481862 us; error 28 us; skew 500 ppm
Aug 10 23:00:37 dist-test-slave-6nmv krb5kdc[20585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:37.484400 20602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:37.485162 20602 webserver.cc:460] Webserver started at http://127.0.0.1:37743/ using document root <none> and password file <none>
I20260810 23:00:37.485687 20602 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:37.485776 20602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:37.486049 20602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:37.487959 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "a66ce38385134145b94a14b07bdbebeb"
format_stamp: "Formatted at 2026-08-10 23:00:37 on dist-test-slave-6nmv"
server_key: "2c39660f10ef92ca39292d39412e4edc"
server_key_iv: "dc8b653b75631f2fbc374644cc6bcd24"
server_key_version: "encryptionkey@0"
I20260810 23:00:37.488549 20602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "a66ce38385134145b94a14b07bdbebeb"
format_stamp: "Formatted at 2026-08-10 23:00:37 on dist-test-slave-6nmv"
server_key: "2c39660f10ef92ca39292d39412e4edc"
server_key_iv: "dc8b653b75631f2fbc374644cc6bcd24"
server_key_version: "encryptionkey@0"
I20260810 23:00:37.491987 20602 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 23:00:37.494038 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:37.495003 20602 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:00:37.495124 20602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "a66ce38385134145b94a14b07bdbebeb"
format_stamp: "Formatted at 2026-08-10 23:00:37 on dist-test-slave-6nmv"
server_key: "2c39660f10ef92ca39292d39412e4edc"
server_key_iv: "dc8b653b75631f2fbc374644cc6bcd24"
server_key_version: "encryptionkey@0"
I20260810 23:00:37.495215 20602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:37.514101 20602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:37.516953 20602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:37.517129 20602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:37.524242 20602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41037
I20260810 23:00:37.524271 20677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41037 every 8 connection(s)
I20260810 23:00:37.525072 20602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:37.527585 20678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:37.530089 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20602
I20260810 23:00:37.530225 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:37.530489 30010 external_mini_cluster.cc:1428] Setting key 06134c253ac5b8e0130307136b0464f6
I20260810 23:00:37.533418 20678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb: Bootstrap starting.
I20260810 23:00:37.535830 20678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:37.536607 20678 log.cc:824] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:37.538493 20678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb: No bootstrap required, opened a new log
Aug 10 23:00:37 dist-test-slave-6nmv krb5kdc[20585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:00:37.542157 20678 raft_consensus.cc:348] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66ce38385134145b94a14b07bdbebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260810 23:00:37.542414 20678 raft_consensus.cc:374] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:37.542488 20678 raft_consensus.cc:729] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a66ce38385134145b94a14b07bdbebeb, State: Initialized, Role: FOLLOWER
I20260810 23:00:37.542914 20678 consensus_queue.cc:260] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66ce38385134145b94a14b07bdbebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260810 23:00:37.543062 20678 raft_consensus.cc:388] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:37.543140 20678 raft_consensus.cc:482] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:37.543252 20678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:37.544052 20678 raft_consensus.cc:504] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66ce38385134145b94a14b07bdbebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260810 23:00:37.544426 20678 leader_election.cc:302] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a66ce38385134145b94a14b07bdbebeb; no voters:
I20260810 23:00:37.544759 20678 leader_election.cc:288] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:37.544857 20683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:37.545030 20683 raft_consensus.cc:686] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [term 1 LEADER]: Becoming Leader. State: Replica: a66ce38385134145b94a14b07bdbebeb, State: Running, Role: LEADER
I20260810 23:00:37.545248 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:37.531755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33613 (local address 127.0.0.1:41037)
0810 23:00:37.532171 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:37.532188 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:37.532234 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:00:37.533039 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:37.533060 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:37.533434 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:37.533752 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:37.534452 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:37.535780 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:37.536648 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:37.537213 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:37.537408 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:37.539750 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:37.539756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:37.539770 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:37.539797 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:37.542344 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:37.542826 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:37.542831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:37.542835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:37.542953 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:37.543263 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:37.543284 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:37.543286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:37.543679 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:37.543879 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:37.544118 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:00:37.544392 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260810 23:00:37.545356 20683 consensus_queue.cc:237] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66ce38385134145b94a14b07bdbebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } }
I20260810 23:00:37.545996 20678 sys_catalog.cc:564] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:37.546556 20683 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a66ce38385134145b94a14b07bdbebeb. Latest consensus state: current_term: 1 leader_uuid: "a66ce38385134145b94a14b07bdbebeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66ce38385134145b94a14b07bdbebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } } }
I20260810 23:00:37.546663 20683 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:37.546958 20684 sys_catalog.cc:455] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a66ce38385134145b94a14b07bdbebeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a66ce38385134145b94a14b07bdbebeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41037 } } }
I20260810 23:00:37.547051 20684 sys_catalog.cc:458] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:37.548229 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:37.548362 20689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:37.551280 20689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:37.556496 20689 catalog_manager.cc:1347] Generated new cluster ID: 729ddded4a5947b6a11c136ceb10a178
I20260810 23:00:37.556551 20689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:37.562096 20681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:37.550451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52967 (local address 127.0.0.1:41037)
0810 23:00:37.550542 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:37.550546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:37.553152 (+ 2606us) server_negotiation.cc:407] Connection header received
0810 23:00:37.553320 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:37.553325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:37.553371 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:37.553456 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:37.554159 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:37.554631 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:37.555457 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:37.555864 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:37.556006 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:37.558796 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:37.558800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:37.558802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:37.558825 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:37.560475 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:37.560893 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:37.560896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:37.560898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:37.560989 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:37.561238 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:37.561241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:37.561242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:37.561468 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:37.561595 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:37.561835 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 23:00:37.561957 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:00:37.574882 20689 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:37.575932 20689 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:37.582436 20689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a66ce38385134145b94a14b07bdbebeb: Generated new TSK 0
I20260810 23:00:37.583086 20689 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:37.588914 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20602
2026-08-10T23:00:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:37 dist-test-slave-6nmv krb5kdc[20713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:37 dist-test-slave-6nmv krb5kdc[20713](info): set up 2 sockets
Aug 10 23:00:37 dist-test-slave-6nmv krb5kdc[20713](info): commencing operation
krb5kdc: starting...
W20260810 23:00:38.635504 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:38 dist-test-slave-6nmv krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:38.754112 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:38.856603 20730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:38.856822 20730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:38.856856 20730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:38.860576 20730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:38.860622 20730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:38.860639 20730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:38.860671 20730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:38.860687 20730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:38.865311 20730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:38.866211 20730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:38.867079 20730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:00:38.875180 20730 server_base.cc:1034] running on GCE node
W20260810 23:00:38.875255 20736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:38.875313 20735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:38.875505 20738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:38.875957 20730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:38.876924 20730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:38.878108 20730 hybrid_clock.cc:648] HybridClock initialized: now 1786402838878090 us; error 27 us; skew 500 ppm
Aug 10 23:00:38 dist-test-slave-6nmv krb5kdc[20713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:38.880648 20730 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:38.881497 20730 webserver.cc:460] Webserver started at http://127.0.0.1:35321/ using document root <none> and password file <none>
I20260810 23:00:38.882083 20730 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:38.882174 20730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:38.882396 20730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:38.884327 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "8b53f333e7c64566bdf8c1498afa0c68"
format_stamp: "Formatted at 2026-08-10 23:00:38 on dist-test-slave-6nmv"
server_key: "6a977ceada662a2996e5854fdfe74d06"
server_key_iv: "5f1419e85052aea7701fc090ff214a99"
server_key_version: "encryptionkey@0"
I20260810 23:00:38.884917 20730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "8b53f333e7c64566bdf8c1498afa0c68"
format_stamp: "Formatted at 2026-08-10 23:00:38 on dist-test-slave-6nmv"
server_key: "6a977ceada662a2996e5854fdfe74d06"
server_key_iv: "5f1419e85052aea7701fc090ff214a99"
server_key_version: "encryptionkey@0"
I20260810 23:00:38.888571 20730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:00:38.891022 20748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:38.892146 20730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:38.892295 20730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "8b53f333e7c64566bdf8c1498afa0c68"
format_stamp: "Formatted at 2026-08-10 23:00:38 on dist-test-slave-6nmv"
server_key: "6a977ceada662a2996e5854fdfe74d06"
server_key_iv: "5f1419e85052aea7701fc090ff214a99"
server_key_version: "encryptionkey@0"
I20260810 23:00:38.892405 20730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:38.914979 20730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:38.917949 20730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:38.918174 20730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:38.925503 20730 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38575
I20260810 23:00:38.925513 20804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38575 every 8 connection(s)
I20260810 23:00:38.926522 20730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:38.929247 20805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:38.929618 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20730
I20260810 23:00:38.929765 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:38.930112 30010 external_mini_cluster.cc:1428] Setting key 40bd56c0f04c0003bccfaf65f5cd672c
I20260810 23:00:38.936151 20805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68: Bootstrap starting.
I20260810 23:00:38.938743 20805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:38.939571 20805 log.cc:824] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:38.939764 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:38.931498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52065 (local address 127.0.0.1:38575)
0810 23:00:38.931942 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:38.931961 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:38.932006 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:00:38.932742 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:38.932764 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:38.933200 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:38.933550 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:38.933963 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:38.935412 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:38.936243 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:38.936841 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:38.936993 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:38.937839 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:38.937846 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:38.937861 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:38.938324 (+ 463us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:38.938402 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:38.938431 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:38.938524 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:38.938678 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 23:00:38.938878 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":122,"threads_started":1}
I20260810 23:00:38.941552 20805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68: No bootstrap required, opened a new log
I20260810 23:00:38.942432 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:38.945540 20805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b53f333e7c64566bdf8c1498afa0c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260810 23:00:38.945763 20805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:38.945847 20805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b53f333e7c64566bdf8c1498afa0c68, State: Initialized, Role: FOLLOWER
I20260810 23:00:38.946301 20805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b53f333e7c64566bdf8c1498afa0c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260810 23:00:38.946456 20805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:38.946517 20805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:38.946609 20805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:38.947750 20805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b53f333e7c64566bdf8c1498afa0c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260810 23:00:38.948132 20805 leader_election.cc:302] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b53f333e7c64566bdf8c1498afa0c68; no voters:
I20260810 23:00:38.948423 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:38.944420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60935 (local address 127.0.0.1:38575)
0810 23:00:38.944534 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:38.944538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:38.944685 (+ 147us) server_negotiation.cc:407] Connection header received
0810 23:00:38.944803 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:38.944805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:38.944841 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:38.944923 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:38.945232 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:38.945666 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:38.946739 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:38.947005 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:38.947087 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:38.947935 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:38.947939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:38.947942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:38.947954 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:38.947989 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:38.948000 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:38.948063 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:38.948183 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:00:38.948225 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:00:38.948488 20805 leader_election.cc:288] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:38.948562 20816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:38.948760 20816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [term 1 LEADER]: Becoming Leader. State: Replica: 8b53f333e7c64566bdf8c1498afa0c68, State: Running, Role: LEADER
I20260810 23:00:38.949070 20816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b53f333e7c64566bdf8c1498afa0c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } }
I20260810 23:00:38.949735 20805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:38.950261 20817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b53f333e7c64566bdf8c1498afa0c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b53f333e7c64566bdf8c1498afa0c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } } }
I20260810 23:00:38.950310 20818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b53f333e7c64566bdf8c1498afa0c68. Latest consensus state: current_term: 1 leader_uuid: "8b53f333e7c64566bdf8c1498afa0c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b53f333e7c64566bdf8c1498afa0c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38575 } } }
I20260810 23:00:38.950385 20817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:38.950423 20818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:38.951831 20821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:38.954376 20821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:38.959390 20821 catalog_manager.cc:1347] Generated new cluster ID: 9edad41563ca4934a53d5bd8950f5207
I20260810 23:00:38.959456 20821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:38.967636 20821 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:38.968458 20821 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:38.974697 20821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b53f333e7c64566bdf8c1498afa0c68: Generated new TSK 0
I20260810 23:00:38.975303 20821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:38.987025 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20730
2026-08-10T23:00:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:39 dist-test-slave-6nmv krb5kdc[20840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:39 dist-test-slave-6nmv krb5kdc[20840](info): set up 2 sockets
Aug 10 23:00:39 dist-test-slave-6nmv krb5kdc[20840](info): commencing operation
krb5kdc: starting...
W20260810 23:00:40.040603 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:40 dist-test-slave-6nmv krb5kdc[20840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:40.164494 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:40.272004 20853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:40.272287 20853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:40.272364 20853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:40.276410 20853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:40.276508 20853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:40.276547 20853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:40.276607 20853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:40.276647 20853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:40.281893 20853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:40.282963 20853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:40.283993 20853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:40.292341 20861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:40.292402 20864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:40.292441 20853 server_base.cc:1034] running on GCE node
W20260810 23:00:40.292425 20859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:40.293197 20853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:40.294318 20853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:40.295482 20853 hybrid_clock.cc:648] HybridClock initialized: now 1786402840295466 us; error 30 us; skew 500 ppm
Aug 10 23:00:40 dist-test-slave-6nmv krb5kdc[20840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402840, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:40.298157 20853 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:40.298928 20853 webserver.cc:460] Webserver started at http://127.0.0.1:43481/ using document root <none> and password file <none>
I20260810 23:00:40.299392 20853 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:40.299455 20853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:40.299605 20853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:40.301483 20853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "4dd52ec7fba046098dc0faf16efbb57e"
format_stamp: "Formatted at 2026-08-10 23:00:40 on dist-test-slave-6nmv"
server_key: "f6b9abf3d34518b76cae199dfe49712a"
server_key_iv: "06afcef13ae575f70721388f78aa9ea6"
server_key_version: "encryptionkey@0"
I20260810 23:00:40.302107 20853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "4dd52ec7fba046098dc0faf16efbb57e"
format_stamp: "Formatted at 2026-08-10 23:00:40 on dist-test-slave-6nmv"
server_key: "f6b9abf3d34518b76cae199dfe49712a"
server_key_iv: "06afcef13ae575f70721388f78aa9ea6"
server_key_version: "encryptionkey@0"
I20260810 23:00:40.305477 20853 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:00:40.307899 20871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:40.309165 20853 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:40.309283 20853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "4dd52ec7fba046098dc0faf16efbb57e"
format_stamp: "Formatted at 2026-08-10 23:00:40 on dist-test-slave-6nmv"
server_key: "f6b9abf3d34518b76cae199dfe49712a"
server_key_iv: "06afcef13ae575f70721388f78aa9ea6"
server_key_version: "encryptionkey@0"
I20260810 23:00:40.309379 20853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:40.313778 20853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:40.316527 20853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:40.316709 20853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:40.323782 20853 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33595
I20260810 23:00:40.323793 20928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33595 every 8 connection(s)
I20260810 23:00:40.324638 20853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:40.327235 20929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:40.330332 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20853
I20260810 23:00:40.330502 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:40.330749 30010 external_mini_cluster.cc:1428] Setting key dc9381d9f96f329d468433b7d4635b00
I20260810 23:00:40.333000 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e: Bootstrap starting.
I20260810 23:00:40.335425 20929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:40.335579 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:40.331979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56367 (local address 127.0.0.1:33595)
0810 23:00:40.332381 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:40.332396 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:40.332431 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:00:40.333025 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:40.333039 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:40.333331 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:40.333586 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:40.334431 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:40.334436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:40.334447 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:40.334457 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:40.334506 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:40.334524 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:40.334628 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:40.334736 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 23:00:40.334895 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":144,"threads_started":1}
I20260810 23:00:40.336158 20929 log.cc:824] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:40.337908 20929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e: No bootstrap required, opened a new log
I20260810 23:00:40.338030 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:40.341679 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:40.340146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46467 (local address 127.0.0.1:33595)
0810 23:00:40.340250 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:40.340254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:40.340470 (+ 216us) server_negotiation.cc:407] Connection header received
0810 23:00:40.340562 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:40.340564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:40.340589 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:40.340644 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:40.341277 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:40.341279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:40.341281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:40.341290 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:40.341310 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:40.341319 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:40.341379 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:40.341514 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:00:40.341542 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:00:40.342337 20929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd52ec7fba046098dc0faf16efbb57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260810 23:00:40.342549 20929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:40.342602 20929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dd52ec7fba046098dc0faf16efbb57e, State: Initialized, Role: FOLLOWER
I20260810 23:00:40.342945 20929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd52ec7fba046098dc0faf16efbb57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260810 23:00:40.343076 20929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:40.343163 20929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:40.343246 20929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:40.343988 20929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd52ec7fba046098dc0faf16efbb57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260810 23:00:40.344288 20929 leader_election.cc:302] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dd52ec7fba046098dc0faf16efbb57e; no voters:
I20260810 23:00:40.344550 20929 leader_election.cc:288] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:40.344661 20941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:40.344878 20941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [term 1 LEADER]: Becoming Leader. State: Replica: 4dd52ec7fba046098dc0faf16efbb57e, State: Running, Role: LEADER
I20260810 23:00:40.345232 20941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd52ec7fba046098dc0faf16efbb57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } }
I20260810 23:00:40.345676 20929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:40.346885 20942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dd52ec7fba046098dc0faf16efbb57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd52ec7fba046098dc0faf16efbb57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260810 23:00:40.346886 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dd52ec7fba046098dc0faf16efbb57e. Latest consensus state: current_term: 1 leader_uuid: "4dd52ec7fba046098dc0faf16efbb57e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dd52ec7fba046098dc0faf16efbb57e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33595 } } }
I20260810 23:00:40.347038 20942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:40.347123 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e [sys.catalog]: This master's current role is: LEADER
W20260810 23:00:40.350260 20963 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:00:40.350327 20963 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:00:40.350404 20957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:40.351127 20957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:40.355481 20957 catalog_manager.cc:1347] Generated new cluster ID: 915ae2d3cc464a2b8207ce91da28f88b
I20260810 23:00:40.355536 20957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:40.371788 20957 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:40.372100 20957 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:40.377691 20957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dd52ec7fba046098dc0faf16efbb57e: Generated new TSK 0
I20260810 23:00:40.378414 20957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:40.389073 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20853
2026-08-10T23:00:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:40 dist-test-slave-6nmv krb5kdc[20968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:40 dist-test-slave-6nmv krb5kdc[20968](info): set up 2 sockets
Aug 10 23:00:40 dist-test-slave-6nmv krb5kdc[20968](info): commencing operation
krb5kdc: starting...
W20260810 23:00:41.463426 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:41 dist-test-slave-6nmv krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:41.584708 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:41.688506 20981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:41.688807 20981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:41.688887 20981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:41.692828 20981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:41.692914 20981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:41.692953 20981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:41.693011 20981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:41.693050 20981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:41.697829 20981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:41.698834 20981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:41.699746 20981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:41.707906 20987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:41.707984 20989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:41.708061 20981 server_base.cc:1034] running on GCE node
W20260810 23:00:41.707893 20991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:41.708727 20981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:41.709723 20981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:41.710906 20981 hybrid_clock.cc:648] HybridClock initialized: now 1786402841710887 us; error 29 us; skew 500 ppm
Aug 10 23:00:41 dist-test-slave-6nmv krb5kdc[20968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:41.718674 20981 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:41.719554 20981 webserver.cc:460] Webserver started at http://127.0.0.1:45801/ using document root <none> and password file <none>
I20260810 23:00:41.720052 20981 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:41.720109 20981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:41.720264 20981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:41.722177 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "d8e271b702d34bb88b1883cb0fb12bcc"
format_stamp: "Formatted at 2026-08-10 23:00:41 on dist-test-slave-6nmv"
server_key: "a6d283b4f809a410dadbc0affa48e814"
server_key_iv: "c4cb7118a8accbbf8d7a1fd8e85b6302"
server_key_version: "encryptionkey@0"
I20260810 23:00:41.722653 20981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "d8e271b702d34bb88b1883cb0fb12bcc"
format_stamp: "Formatted at 2026-08-10 23:00:41 on dist-test-slave-6nmv"
server_key: "a6d283b4f809a410dadbc0affa48e814"
server_key_iv: "c4cb7118a8accbbf8d7a1fd8e85b6302"
server_key_version: "encryptionkey@0"
I20260810 23:00:41.725950 20981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:00:41.728063 20998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:41.729313 20981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:41.729427 20981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "d8e271b702d34bb88b1883cb0fb12bcc"
format_stamp: "Formatted at 2026-08-10 23:00:41 on dist-test-slave-6nmv"
server_key: "a6d283b4f809a410dadbc0affa48e814"
server_key_iv: "c4cb7118a8accbbf8d7a1fd8e85b6302"
server_key_version: "encryptionkey@0"
I20260810 23:00:41.729532 20981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:41.744307 20981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:41.746935 20981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:41.747076 20981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:41.754134 20981 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43663
I20260810 23:00:41.754158 21055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43663 every 8 connection(s)
I20260810 23:00:41.755019 20981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:41.757781 21056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:41.760288 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 20981
I20260810 23:00:41.760434 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:41.760720 30010 external_mini_cluster.cc:1428] Setting key 8cf8a99ed2238e3af0f1ea85d062c23e
I20260810 23:00:41.764359 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc: Bootstrap starting.
I20260810 23:00:41.767059 21056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:41.767974 21056 log.cc:824] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:41.770108 21056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc: No bootstrap required, opened a new log
Aug 10 23:00:41 dist-test-slave-6nmv krb5kdc[20968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:00:41.774466 21056 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260810 23:00:41.774792 21056 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:41.774895 21056 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8e271b702d34bb88b1883cb0fb12bcc, State: Initialized, Role: FOLLOWER
I20260810 23:00:41.775353 21056 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260810 23:00:41.775506 21056 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:41.775583 21056 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:41.775686 21056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:41.776726 21056 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260810 23:00:41.776770 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:41.762239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:41805 (local address 127.0.0.1:43663)
0810 23:00:41.762640 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:41.762659 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:41.762704 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:00:41.763542 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:41.763563 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:41.763987 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:41.764350 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:41.765178 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:41.766682 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:41.767546 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:41.768251 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:41.768442 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:41.771041 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:41.771047 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:41.771062 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:41.771094 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:41.773886 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:41.774376 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:41.774382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:41.774387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:41.774537 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:41.774937 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:41.774942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:41.774946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:41.775329 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:41.775582 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:41.775858 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 23:00:41.776069 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":104,"threads_started":1}
I20260810 23:00:41.777117 21056 leader_election.cc:302] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8e271b702d34bb88b1883cb0fb12bcc; no voters:
I20260810 23:00:41.777448 21056 leader_election.cc:288] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:41.777559 21061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:41.777770 21061 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [term 1 LEADER]: Becoming Leader. State: Replica: d8e271b702d34bb88b1883cb0fb12bcc, State: Running, Role: LEADER
I20260810 23:00:41.778155 21061 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } }
I20260810 23:00:41.778738 21056 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:41.779384 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8e271b702d34bb88b1883cb0fb12bcc. Latest consensus state: current_term: 1 leader_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } } }
I20260810 23:00:41.780120 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:41.780197 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:41.780068 21062 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8e271b702d34bb88b1883cb0fb12bcc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43663 } } }
I20260810 23:00:41.780431 21062 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:41.781086 21070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:41.784065 21070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:41.789690 21070 catalog_manager.cc:1347] Generated new cluster ID: 58bac3f6942b4673aa5964a17e88b99c
I20260810 23:00:41.789788 21070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:41.790606 21059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:41.782396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:55069 (local address 127.0.0.1:43663)
0810 23:00:41.782517 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:41.782520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:41.782740 (+ 220us) server_negotiation.cc:407] Connection header received
0810 23:00:41.782900 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:41.782903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:41.782963 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:41.783042 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:41.783651 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:41.784122 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:41.784866 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:41.785097 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:41.785203 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:41.786494 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:41.786498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:41.786500 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:41.786521 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:41.788663 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:41.789250 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:41.789253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:41.789255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:41.789336 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:41.789715 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:41.789718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:41.789722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:41.789949 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:41.790088 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:41.790382 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 23:00:41.790475 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:00:41.801923 21070 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:41.802651 21070 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:41.814119 21070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8e271b702d34bb88b1883cb0fb12bcc: Generated new TSK 0
I20260810 23:00:41.814811 21070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:41.818414 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 20981
W20260810 23:00:41.826984 20966 connection.cc:535] client connection to 127.0.0.1:43663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43663): Connection reset by peer (error 104)
2026-08-10T23:00:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:41 dist-test-slave-6nmv krb5kdc[21089](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:41 dist-test-slave-6nmv krb5kdc[21089](info): set up 2 sockets
Aug 10 23:00:41 dist-test-slave-6nmv krb5kdc[21089](info): commencing operation
krb5kdc: starting...
W20260810 23:00:42.875648 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:42 dist-test-slave-6nmv krb5kdc[21089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:42.990901 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:43.097976 21103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:43.098222 21103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:43.098263 21103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:43.101961 21103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:43.102003 21103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:43.102021 21103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:43.102036 21103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:43.102051 21103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:43.106950 21103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:43.107858 21103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:43.108788 21103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:43.116709 21111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:43.116706 21110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:43.116711 21114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:43.116822 21103 server_base.cc:1034] running on GCE node
I20260810 23:00:43.117468 21103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:43.118549 21103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:43.119740 21103 hybrid_clock.cc:648] HybridClock initialized: now 1786402843119713 us; error 38 us; skew 500 ppm
Aug 10 23:00:43 dist-test-slave-6nmv krb5kdc[21089](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:43.122432 21103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:43.123219 21103 webserver.cc:460] Webserver started at http://127.0.0.1:41961/ using document root <none> and password file <none>
I20260810 23:00:43.123716 21103 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:43.123822 21103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:43.124061 21103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:43.125911 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "c97e6512d31747b9b00282775a8aae7b"
format_stamp: "Formatted at 2026-08-10 23:00:43 on dist-test-slave-6nmv"
server_key: "0fc498f290f32eb8d4611d238500b77d"
server_key_iv: "447188c0cb310e62651ed861c0bff761"
server_key_version: "encryptionkey@0"
I20260810 23:00:43.126461 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "c97e6512d31747b9b00282775a8aae7b"
format_stamp: "Formatted at 2026-08-10 23:00:43 on dist-test-slave-6nmv"
server_key: "0fc498f290f32eb8d4611d238500b77d"
server_key_iv: "447188c0cb310e62651ed861c0bff761"
server_key_version: "encryptionkey@0"
I20260810 23:00:43.130513 21103 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260810 23:00:43.132763 21122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:43.133904 21103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:43.134032 21103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "c97e6512d31747b9b00282775a8aae7b"
format_stamp: "Formatted at 2026-08-10 23:00:43 on dist-test-slave-6nmv"
server_key: "0fc498f290f32eb8d4611d238500b77d"
server_key_iv: "447188c0cb310e62651ed861c0bff761"
server_key_version: "encryptionkey@0"
I20260810 23:00:43.134131 21103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:43.146184 21103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:43.148684 21103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:43.148892 21103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:43.155683 21103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37969
I20260810 23:00:43.155723 21180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37969 every 8 connection(s)
I20260810 23:00:43.156543 21103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:43.159116 21181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:43.164225 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b: Bootstrap starting.
I20260810 23:00:43.166528 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21103
I20260810 23:00:43.166654 21181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:43.166669 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:43.166992 30010 external_mini_cluster.cc:1428] Setting key 25eeb2d8bad90492fe4b3709af2a9d57
I20260810 23:00:43.167497 21181 log.cc:824] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:43.169343 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b: No bootstrap required, opened a new log
I20260810 23:00:43.173475 21181 raft_consensus.cc:348] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97e6512d31747b9b00282775a8aae7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260810 23:00:43.173745 21181 raft_consensus.cc:374] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:43.173880 21181 raft_consensus.cc:729] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97e6512d31747b9b00282775a8aae7b, State: Initialized, Role: FOLLOWER
I20260810 23:00:43.174304 21181 consensus_queue.cc:260] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97e6512d31747b9b00282775a8aae7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260810 23:00:43.174428 21181 raft_consensus.cc:388] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:43.174513 21181 raft_consensus.cc:482] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:43.174633 21181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:43.175503 21181 raft_consensus.cc:504] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97e6512d31747b9b00282775a8aae7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260810 23:00:43.175737 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:43.168075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:42679 (local address 127.0.0.1:37969)
0810 23:00:43.168512 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:43.168527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:43.168560 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:00:43.169155 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:43.169169 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:43.169516 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:43.169768 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:43.170168 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:43.171099 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:43.172453 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:43.172888 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:43.173001 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:43.173904 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:43.173908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:43.173919 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:43.174358 (+ 439us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:43.174446 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:43.174485 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:43.174638 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:43.174876 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 23:00:43.175011 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":154,"threads_started":1}
I20260810 23:00:43.175840 21181 leader_election.cc:302] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c97e6512d31747b9b00282775a8aae7b; no voters:
I20260810 23:00:43.176151 21181 leader_election.cc:288] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:43.176244 21187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:43.176455 21187 raft_consensus.cc:686] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [term 1 LEADER]: Becoming Leader. State: Replica: c97e6512d31747b9b00282775a8aae7b, State: Running, Role: LEADER
I20260810 23:00:43.176712 21187 consensus_queue.cc:237] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97e6512d31747b9b00282775a8aae7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } }
I20260810 23:00:43.177217 21181 sys_catalog.cc:564] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:43.177866 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c97e6512d31747b9b00282775a8aae7b. Latest consensus state: current_term: 1 leader_uuid: "c97e6512d31747b9b00282775a8aae7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97e6512d31747b9b00282775a8aae7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } } }
I20260810 23:00:43.177978 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:43.178371 21194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:43.178391 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c97e6512d31747b9b00282775a8aae7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c97e6512d31747b9b00282775a8aae7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37969 } } }
I20260810 23:00:43.178507 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:43.178874 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:43.181344 21194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:43.186439 21194 catalog_manager.cc:1347] Generated new cluster ID: 9853e8e72f684abfb5aef40147e91bb2
I20260810 23:00:43.186497 21194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:43.187851 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:43.182252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:52587 (local address 127.0.0.1:37969)
0810 23:00:43.182369 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:43.182373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:43.183360 (+ 987us) server_negotiation.cc:407] Connection header received
0810 23:00:43.183473 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:43.183476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:43.183522 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:43.183606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:43.183885 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:43.184651 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:43.185840 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:43.186198 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:43.186276 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:43.187366 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:43.187370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:43.187372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:43.187384 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:43.187408 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:43.187419 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:43.187501 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:43.187676 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 23:00:43.187706 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":47}
I20260810 23:00:43.231868 21194 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:43.232810 21194 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:43.237195 21194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c97e6512d31747b9b00282775a8aae7b: Generated new TSK 0
I20260810 23:00:43.237955 21194 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:43.242350 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21103
W20260810 23:00:43.251744 21087 connection.cc:535] client connection to 127.0.0.1:37969 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37969): Connection reset by peer (error 104)
2026-08-10T23:00:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:43 dist-test-slave-6nmv krb5kdc[21216](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:43 dist-test-slave-6nmv krb5kdc[21216](info): set up 2 sockets
Aug 10 23:00:43 dist-test-slave-6nmv krb5kdc[21216](info): commencing operation
krb5kdc: starting...
W20260810 23:00:44.298944 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:44 dist-test-slave-6nmv krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:44.420768 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:44.528311 21229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:44.528604 21229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:44.528678 21229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:44.532572 21229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:44.532657 21229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:44.532694 21229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:44.532749 21229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:44.532797 21229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:44.537431 21229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:44.538494 21229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:44.539470 21229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:44.547765 21235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:44.547858 21237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:44.547747 21234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:44.548027 21229 server_base.cc:1034] running on GCE node
I20260810 23:00:44.548604 21229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:44.549685 21229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:44.550844 21229 hybrid_clock.cc:648] HybridClock initialized: now 1786402844550828 us; error 29 us; skew 500 ppm
Aug 10 23:00:44 dist-test-slave-6nmv krb5kdc[21216](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:44.553483 21229 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:44.554337 21229 webserver.cc:460] Webserver started at http://127.0.0.1:33815/ using document root <none> and password file <none>
I20260810 23:00:44.554843 21229 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:44.554931 21229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:44.555140 21229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:44.557019 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "655cabf346804de8b62e937a01c711f1"
format_stamp: "Formatted at 2026-08-10 23:00:44 on dist-test-slave-6nmv"
server_key: "6fff14c99d465ffdb10d04c02b31b881"
server_key_iv: "f1a20c4fae214f0e11c250c5bdf035b2"
server_key_version: "encryptionkey@0"
I20260810 23:00:44.557575 21229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "655cabf346804de8b62e937a01c711f1"
format_stamp: "Formatted at 2026-08-10 23:00:44 on dist-test-slave-6nmv"
server_key: "6fff14c99d465ffdb10d04c02b31b881"
server_key_iv: "f1a20c4fae214f0e11c250c5bdf035b2"
server_key_version: "encryptionkey@0"
I20260810 23:00:44.560949 21229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 23:00:44.563153 21247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:44.564180 21229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:44.564309 21229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "655cabf346804de8b62e937a01c711f1"
format_stamp: "Formatted at 2026-08-10 23:00:44 on dist-test-slave-6nmv"
server_key: "6fff14c99d465ffdb10d04c02b31b881"
server_key_iv: "f1a20c4fae214f0e11c250c5bdf035b2"
server_key_version: "encryptionkey@0"
I20260810 23:00:44.564406 21229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:44.568794 21229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:44.571564 21229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:44.571761 21229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:44.578909 21229 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40097
I20260810 23:00:44.578946 21305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40097 every 8 connection(s)
I20260810 23:00:44.579767 21229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:44.582361 21306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:44.586503 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21229
I20260810 23:00:44.586640 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:44.586920 30010 external_mini_cluster.cc:1428] Setting key 45d53ee3b76c75d79b272eea011b92ab
I20260810 23:00:44.587615 21306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1: Bootstrap starting.
I20260810 23:00:44.590004 21306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:44.590852 21306 log.cc:824] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:44.592307 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:44.588292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:34705 (local address 127.0.0.1:40097)
0810 23:00:44.588723 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:44.588737 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:44.588773 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:00:44.589370 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:44.589385 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:44.589646 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:44.590061 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:44.590838 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:44.590843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:44.590863 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:44.590874 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:44.590935 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:44.590954 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:44.591079 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:44.591232 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:00:44.591563 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260810 23:00:44.592854 21306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1: No bootstrap required, opened a new log
I20260810 23:00:44.594880 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:44.596520 21306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655cabf346804de8b62e937a01c711f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260810 23:00:44.596751 21306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:44.596865 21306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 655cabf346804de8b62e937a01c711f1, State: Initialized, Role: FOLLOWER
I20260810 23:00:44.597327 21306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655cabf346804de8b62e937a01c711f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260810 23:00:44.597492 21306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:44.597570 21306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:44.597684 21306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:44.598562 21309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:44.596772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:32849 (local address 127.0.0.1:40097)
0810 23:00:44.596896 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:44.596899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:44.597034 (+ 135us) server_negotiation.cc:407] Connection header received
0810 23:00:44.597177 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:44.597180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:44.597217 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:44.597277 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:44.598196 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:44.598199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:44.598201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:44.598211 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:44.598233 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:44.598243 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:44.598312 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:44.598410 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 23:00:44.598439 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:00:44.599072 21306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655cabf346804de8b62e937a01c711f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260810 23:00:44.599424 21306 leader_election.cc:302] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 655cabf346804de8b62e937a01c711f1; no voters:
I20260810 23:00:44.599807 21306 leader_election.cc:288] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:44.599948 21318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:44.600176 21318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [term 1 LEADER]: Becoming Leader. State: Replica: 655cabf346804de8b62e937a01c711f1, State: Running, Role: LEADER
I20260810 23:00:44.600502 21318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655cabf346804de8b62e937a01c711f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } }
I20260810 23:00:44.601233 21306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:44.601851 21318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 655cabf346804de8b62e937a01c711f1. Latest consensus state: current_term: 1 leader_uuid: "655cabf346804de8b62e937a01c711f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655cabf346804de8b62e937a01c711f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260810 23:00:44.601975 21318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:44.602417 21322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:44.602726 21319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "655cabf346804de8b62e937a01c711f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "655cabf346804de8b62e937a01c711f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40097 } } }
I20260810 23:00:44.602837 21319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:44.605494 21322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:44.614560 21322 catalog_manager.cc:1347] Generated new cluster ID: d25a8e56d1a744b5946d9c7135d99987
I20260810 23:00:44.614640 21322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:44.629341 21322 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:44.629688 21322 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:44.636480 21322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 655cabf346804de8b62e937a01c711f1: Generated new TSK 0
I20260810 23:00:44.637604 21322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:44.649469 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21229
2026-08-10T23:00:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:44 dist-test-slave-6nmv krb5kdc[21341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:44 dist-test-slave-6nmv krb5kdc[21341](info): set up 2 sockets
Aug 10 23:00:44 dist-test-slave-6nmv krb5kdc[21341](info): commencing operation
krb5kdc: starting...
W20260810 23:00:45.701988 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:45 dist-test-slave-6nmv krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:45.820976 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:45.921826 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:45.922170 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:45.922248 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:45.928565 21354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:45.928618 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:45.928638 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:45.928654 21354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:45.928669 21354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:45.933629 21354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:45.934551 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:45.935431 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:45.943751 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:45.943810 21354 server_base.cc:1034] running on GCE node
W20260810 23:00:45.943815 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:45.944031 21363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:45.944516 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:45.945585 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:45.946766 21354 hybrid_clock.cc:648] HybridClock initialized: now 1786402845946751 us; error 35 us; skew 500 ppm
Aug 10 23:00:45 dist-test-slave-6nmv krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402845, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:45.949479 21354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:45.950367 21354 webserver.cc:460] Webserver started at http://127.0.0.1:43025/ using document root <none> and password file <none>
I20260810 23:00:45.950914 21354 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:45.951009 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:45.951236 21354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:45.953223 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "449bc9a4e23f48c28ae31c0a3052f137"
format_stamp: "Formatted at 2026-08-10 23:00:45 on dist-test-slave-6nmv"
server_key: "2c4a873582347098bcd253b440c82f4c"
server_key_iv: "8fb37fc06274d06fabc6fe1d678dde3b"
server_key_version: "encryptionkey@0"
I20260810 23:00:45.953807 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "449bc9a4e23f48c28ae31c0a3052f137"
format_stamp: "Formatted at 2026-08-10 23:00:45 on dist-test-slave-6nmv"
server_key: "2c4a873582347098bcd253b440c82f4c"
server_key_iv: "8fb37fc06274d06fabc6fe1d678dde3b"
server_key_version: "encryptionkey@0"
I20260810 23:00:45.957507 21354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:00:45.959961 21370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:45.961122 21354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:45.961262 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "449bc9a4e23f48c28ae31c0a3052f137"
format_stamp: "Formatted at 2026-08-10 23:00:45 on dist-test-slave-6nmv"
server_key: "2c4a873582347098bcd253b440c82f4c"
server_key_iv: "8fb37fc06274d06fabc6fe1d678dde3b"
server_key_version: "encryptionkey@0"
I20260810 23:00:45.961372 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:45.973376 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:45.975967 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:45.976162 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:45.982822 21354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39753
I20260810 23:00:45.982841 21435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39753 every 8 connection(s)
I20260810 23:00:45.983671 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:45.986168 21436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:45.987090 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21354
I20260810 23:00:45.987247 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:45.987509 30010 external_mini_cluster.cc:1428] Setting key 0660ad1fa81e5ab296f8799e6ae20566
I20260810 23:00:45.992483 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137: Bootstrap starting.
I20260810 23:00:45.994843 21436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:45.995501 21436 log.cc:824] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:00:45 dist-test-slave-6nmv krb5kdc[21341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:00:45.997360 21436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137: No bootstrap required, opened a new log
I20260810 23:00:46.001106 21436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449bc9a4e23f48c28ae31c0a3052f137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260810 23:00:46.001317 21436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:46.001390 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:45.988867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:49421 (local address 127.0.0.1:39753)
0810 23:00:45.989290 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:45.989307 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:45.989345 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:00:45.990144 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:45.990163 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:45.990571 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:45.990896 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:45.991509 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:45.992920 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:45.993672 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:45.994306 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:45.994442 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:45.996915 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:45.996919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:45.996930 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:45.996954 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:45.998653 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:45.999256 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:45.999260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:45.999263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:45.999357 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:45.999704 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:45.999705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:45.999706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:46.000035 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:46.000184 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:46.000476 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 23:00:46.000683 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":99,"threads_started":1}
I20260810 23:00:46.001394 21436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449bc9a4e23f48c28ae31c0a3052f137, State: Initialized, Role: FOLLOWER
I20260810 23:00:46.001802 21436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449bc9a4e23f48c28ae31c0a3052f137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260810 23:00:46.001986 21436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:46.002061 21436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:46.002146 21436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:46.002918 21436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449bc9a4e23f48c28ae31c0a3052f137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260810 23:00:46.003260 21436 leader_election.cc:302] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 449bc9a4e23f48c28ae31c0a3052f137; no voters:
I20260810 23:00:46.003568 21436 leader_election.cc:288] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:46.003643 21442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:46.003818 21442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [term 1 LEADER]: Becoming Leader. State: Replica: 449bc9a4e23f48c28ae31c0a3052f137, State: Running, Role: LEADER
I20260810 23:00:46.004056 21442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449bc9a4e23f48c28ae31c0a3052f137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } }
I20260810 23:00:46.004186 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:46.004637 21436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:46.005434 21444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "449bc9a4e23f48c28ae31c0a3052f137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449bc9a4e23f48c28ae31c0a3052f137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } } }
I20260810 23:00:46.005559 21444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:46.005860 21443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 449bc9a4e23f48c28ae31c0a3052f137. Latest consensus state: current_term: 1 leader_uuid: "449bc9a4e23f48c28ae31c0a3052f137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "449bc9a4e23f48c28ae31c0a3052f137" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39753 } } }
I20260810 23:00:46.006014 21443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:46.006444 21457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:46.009837 21457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:46.016024 21457 catalog_manager.cc:1347] Generated new cluster ID: ec908f6529474c5084db0a69f1a5a9af
I20260810 23:00:46.016132 21457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:46.016650 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:46.006447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:52359 (local address 127.0.0.1:39753)
0810 23:00:46.006573 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:46.006579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:46.007166 (+ 587us) server_negotiation.cc:407] Connection header received
0810 23:00:46.007274 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:46.007276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:46.007323 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:46.007407 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:46.008183 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:46.009171 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:46.010589 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:46.010884 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:46.010984 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:46.012803 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:46.012806 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:46.012809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:46.012834 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:46.014612 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:46.015120 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:46.015123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:46.015125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:46.015204 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:46.015560 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:46.015563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:46.015565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:46.015754 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:46.015880 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:46.016463 (+ 583us) server_negotiation.cc:299] Negotiation successful
0810 23:00:46.016552 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:00:46.038134 21457 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:46.039332 21457 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:46.048861 21457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 449bc9a4e23f48c28ae31c0a3052f137: Generated new TSK 0
I20260810 23:00:46.049669 21457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:46.061513 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21354
W20260810 23:00:46.072402 21339 connection.cc:535] client connection to 127.0.0.1:39753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39753): Connection reset by peer (error 104)
2026-08-10T23:00:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:46 dist-test-slave-6nmv krb5kdc[21471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:46 dist-test-slave-6nmv krb5kdc[21471](info): set up 2 sockets
Aug 10 23:00:46 dist-test-slave-6nmv krb5kdc[21471](info): commencing operation
krb5kdc: starting...
W20260810 23:00:47.102774 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:47 dist-test-slave-6nmv krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:47.214180 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42025
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:47.319235 21485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:47.319463 21485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:47.319507 21485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:47.323166 21485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:47.323211 21485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:47.323230 21485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:47.323243 21485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:47.323293 21485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:47.327950 21485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42025
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:47.328927 21485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:47.329869 21485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:47.337515 21491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:47.337565 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:47.337594 21492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:47.337677 21485 server_base.cc:1034] running on GCE node
I20260810 23:00:47.338321 21485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:47.339421 21485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:47.340598 21485 hybrid_clock.cc:648] HybridClock initialized: now 1786402847340574 us; error 37 us; skew 500 ppm
Aug 10 23:00:47 dist-test-slave-6nmv krb5kdc[21471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:47.347450 21485 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:47.348264 21485 webserver.cc:460] Webserver started at http://127.0.0.1:40625/ using document root <none> and password file <none>
I20260810 23:00:47.348785 21485 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:47.348873 21485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:47.349117 21485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:47.351006 21485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "33b4659bbcb845fda6855e8e7bb6e3bb"
format_stamp: "Formatted at 2026-08-10 23:00:47 on dist-test-slave-6nmv"
server_key: "c2fcbc4b917ec4e0ac91d4913e33edfb"
server_key_iv: "6a3f0e04c4efb3141b051561c5313ece"
server_key_version: "encryptionkey@0"
I20260810 23:00:47.351459 21485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "33b4659bbcb845fda6855e8e7bb6e3bb"
format_stamp: "Formatted at 2026-08-10 23:00:47 on dist-test-slave-6nmv"
server_key: "c2fcbc4b917ec4e0ac91d4913e33edfb"
server_key_iv: "6a3f0e04c4efb3141b051561c5313ece"
server_key_version: "encryptionkey@0"
I20260810 23:00:47.354743 21485 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:00:47.356935 21504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:47.358093 21485 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:47.358203 21485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "33b4659bbcb845fda6855e8e7bb6e3bb"
format_stamp: "Formatted at 2026-08-10 23:00:47 on dist-test-slave-6nmv"
server_key: "c2fcbc4b917ec4e0ac91d4913e33edfb"
server_key_iv: "6a3f0e04c4efb3141b051561c5313ece"
server_key_version: "encryptionkey@0"
I20260810 23:00:47.358279 21485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:47.374023 21485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:47.376823 21485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:47.376955 21485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:47.384187 21485 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42025
I20260810 23:00:47.384208 21565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42025 every 8 connection(s)
I20260810 23:00:47.385047 21485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:47.387638 21566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:47.389638 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21485
I20260810 23:00:47.389796 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:47.390089 30010 external_mini_cluster.cc:1428] Setting key e8d69661bb54eeca86bbfebb1419c7d1
I20260810 23:00:47.394460 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb: Bootstrap starting.
I20260810 23:00:47.396867 21566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:47.397727 21566 log.cc:824] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:47.399595 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:47.391415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:53175 (local address 127.0.0.1:42025)
0810 23:00:47.391839 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:47.391857 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:47.391898 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:00:47.392618 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:47.392640 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:47.393041 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:47.393405 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:47.393789 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:47.395154 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:47.395935 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:47.396626 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:47.396787 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:47.397622 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:47.397629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:47.397642 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:47.398163 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:47.398229 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:47.398252 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:47.398362 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:47.398614 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:00:47.398737 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":121,"threads_started":1}
I20260810 23:00:47.399922 21566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb: No bootstrap required, opened a new log
I20260810 23:00:47.402283 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:47.403741 21566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42025 } }
I20260810 23:00:47.403976 21566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:47.404062 21566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b4659bbcb845fda6855e8e7bb6e3bb, State: Initialized, Role: FOLLOWER
I20260810 23:00:47.404613 21566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42025 } }
I20260810 23:00:47.404776 21566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:47.404881 21566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:47.404984 21566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:47.405850 21566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42025 } }
I20260810 23:00:47.406177 21566 leader_election.cc:302] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b4659bbcb845fda6855e8e7bb6e3bb; no voters:
I20260810 23:00:47.406451 21566 leader_election.cc:288] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:47.406574 21578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:47.406811 21578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [term 1 LEADER]: Becoming Leader. State: Replica: 33b4659bbcb845fda6855e8e7bb6e3bb, State: Running, Role: LEADER
I20260810 23:00:47.407145 21578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42025 } }
I20260810 23:00:47.407702 21566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:47.408214 21578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b4659bbcb845fda6855e8e7bb6e3bb. Latest consensus state: current_term: 1 leader_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42025 } } }
I20260810 23:00:47.408391 21578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:47.408699 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:47.404259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:36641 (local address 127.0.0.1:42025)
0810 23:00:47.404382 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:47.404386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:47.404606 (+ 220us) server_negotiation.cc:407] Connection header received
0810 23:00:47.404710 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:47.404711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:47.404743 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:47.404840 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:47.405151 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:47.405600 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:47.406674 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:47.407018 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:47.407134 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:47.408032 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:47.408036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:47.408038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:47.408050 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:47.408078 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:47.408090 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:47.408197 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:47.408532 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:00:47.408566 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:00:47.408655 21579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b4659bbcb845fda6855e8e7bb6e3bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42025 } } }
I20260810 23:00:47.408986 21587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:47.409679 21579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:47.411924 21587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:47.417167 21587 catalog_manager.cc:1347] Generated new cluster ID: 4490d2e772084664af7ef1017fe80f54
I20260810 23:00:47.417227 21587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:47.437884 21587 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:47.439046 21587 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:47.450485 21587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33b4659bbcb845fda6855e8e7bb6e3bb: Generated new TSK 0
I20260810 23:00:47.451133 21587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:47.454949 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21485
W20260810 23:00:47.461681 21469 connection.cc:535] client connection to 127.0.0.1:42025 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42025): Connection reset by peer (error 104)
2026-08-10T23:00:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:47 dist-test-slave-6nmv krb5kdc[21600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:47 dist-test-slave-6nmv krb5kdc[21600](info): set up 2 sockets
Aug 10 23:00:47 dist-test-slave-6nmv krb5kdc[21600](info): commencing operation
krb5kdc: starting...
W20260810 23:00:48.494177 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:48 dist-test-slave-6nmv krb5kdc[21600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:00:48.610026 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46549
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:48.713287 21615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:48.713553 21615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:48.713593 21615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:48.717425 21615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:48.717480 21615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:48.717499 21615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:48.717514 21615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:48.717528 21615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:48.722438 21615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46549
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:48.723340 21615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:48.724216 21615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:48.732299 21623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:48.732295 21621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:48.732380 21620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:48.732441 21615 server_base.cc:1034] running on GCE node
I20260810 23:00:48.733052 21615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:48.734098 21615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:48.735275 21615 hybrid_clock.cc:648] HybridClock initialized: now 1786402848735240 us; error 47 us; skew 500 ppm
Aug 10 23:00:48 dist-test-slave-6nmv krb5kdc[21600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:48.737994 21615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:00:48.738864 21615 webserver.cc:460] Webserver started at http://127.0.0.1:35635/ using document root <none> and password file <none>
I20260810 23:00:48.739401 21615 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:48.739501 21615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:48.739722 21615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:48.741655 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "b2cc50dabe7c473888b731aaddfaafb4"
format_stamp: "Formatted at 2026-08-10 23:00:48 on dist-test-slave-6nmv"
server_key: "018cb02558a6c39ff886b6afb3120db5"
server_key_iv: "81af1acaa92059a926faacfa6dfe72e0"
server_key_version: "encryptionkey@0"
I20260810 23:00:48.742264 21615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "b2cc50dabe7c473888b731aaddfaafb4"
format_stamp: "Formatted at 2026-08-10 23:00:48 on dist-test-slave-6nmv"
server_key: "018cb02558a6c39ff886b6afb3120db5"
server_key_iv: "81af1acaa92059a926faacfa6dfe72e0"
server_key_version: "encryptionkey@0"
I20260810 23:00:48.745712 21615 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:00:48.747972 21630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:48.749068 21615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:48.749212 21615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "b2cc50dabe7c473888b731aaddfaafb4"
format_stamp: "Formatted at 2026-08-10 23:00:48 on dist-test-slave-6nmv"
server_key: "018cb02558a6c39ff886b6afb3120db5"
server_key_iv: "81af1acaa92059a926faacfa6dfe72e0"
server_key_version: "encryptionkey@0"
I20260810 23:00:48.749321 21615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:48.753737 21615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:48.756824 21615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:48.757007 21615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:48.764649 21615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46549
I20260810 23:00:48.764765 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46549 every 8 connection(s)
I20260810 23:00:48.765664 21615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:48.768311 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:48.773383 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4: Bootstrap starting.
I20260810 23:00:48.775243 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21615
I20260810 23:00:48.775395 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:48.775668 30010 external_mini_cluster.cc:1428] Setting key 2ba69a0f728ce9b5d2ac9c859938279f
I20260810 23:00:48.775915 21692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:48.776733 21692 log.cc:824] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:48.778741 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4: No bootstrap required, opened a new log
I20260810 23:00:48.780201 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:48.776686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:37555 (local address 127.0.0.1:46549)
0810 23:00:48.777116 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:48.777130 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:48.777162 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:00:48.777724 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:48.777737 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:48.778074 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:48.778347 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:48.779116 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:48.779120 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:48.779130 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:48.779140 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:48.779188 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:48.779205 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:48.779288 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:48.779420 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 23:00:48.779601 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":146,"threads_started":1}
I20260810 23:00:48.782516 21692 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cc50dabe7c473888b731aaddfaafb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260810 23:00:48.782785 21692 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:48.782883 21692 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cc50dabe7c473888b731aaddfaafb4, State: Initialized, Role: FOLLOWER
I20260810 23:00:48.782992 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:48.783340 21692 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cc50dabe7c473888b731aaddfaafb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260810 23:00:48.783509 21692 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:48.783574 21692 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:48.783663 21692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:48.784739 21692 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cc50dabe7c473888b731aaddfaafb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260810 23:00:48.785076 21692 leader_election.cc:302] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2cc50dabe7c473888b731aaddfaafb4; no voters:
I20260810 23:00:48.785384 21692 leader_election.cc:288] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:48.785480 21702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:48.785723 21702 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [term 1 LEADER]: Becoming Leader. State: Replica: b2cc50dabe7c473888b731aaddfaafb4, State: Running, Role: LEADER
I20260810 23:00:48.786054 21702 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cc50dabe7c473888b731aaddfaafb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } }
I20260810 23:00:48.786522 21692 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:48.787544 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2cc50dabe7c473888b731aaddfaafb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cc50dabe7c473888b731aaddfaafb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } } }
I20260810 23:00:48.787664 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:48.787973 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2cc50dabe7c473888b731aaddfaafb4. Latest consensus state: current_term: 1 leader_uuid: "b2cc50dabe7c473888b731aaddfaafb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2cc50dabe7c473888b731aaddfaafb4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46549 } } }
I20260810 23:00:48.788105 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:48.788045 21712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:48.790316 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:48.785205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.217:34105 (local address 127.0.0.1:46549)
0810 23:00:48.785335 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:48.785341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:48.788916 (+ 3575us) server_negotiation.cc:407] Connection header received
0810 23:00:48.789045 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:48.789048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:48.789083 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:48.789167 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:48.789929 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:48.789932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:48.789934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:48.789948 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:48.789974 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:48.789986 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:48.790074 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:48.790184 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:00:48.790218 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:00:48.791465 21712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:48.796178 21712 catalog_manager.cc:1347] Generated new cluster ID: fa2ba53012e94346b04e806b651524c7
I20260810 23:00:48.796231 21712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:48.815853 21712 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:48.816143 21712 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:48.824661 21712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2cc50dabe7c473888b731aaddfaafb4: Generated new TSK 0
I20260810 23:00:48.825201 21712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:48.827402 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21615
2026-08-10T23:00:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:48 dist-test-slave-6nmv krb5kdc[21724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:48 dist-test-slave-6nmv krb5kdc[21724](info): set up 2 sockets
Aug 10 23:00:48 dist-test-slave-6nmv krb5kdc[21724](info): commencing operation
krb5kdc: starting...
W20260810 23:00:49.876935 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:49 dist-test-slave-6nmv krb5kdc[21724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:50.012938 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36713
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:36713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:50.125725 21739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:50.126035 21739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:50.126077 21739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:50.129911 21739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:50.129957 21739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:50.129976 21739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:50.129992 21739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:50.130007 21739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:50.134970 21739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:36713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:50.135820 21739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:50.136741 21739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:50.145018 21744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:50.145018 21747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:50.145239 21739 server_base.cc:1034] running on GCE node
W20260810 23:00:50.145190 21745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:50.145970 21739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:50.146957 21739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:50.148152 21739 hybrid_clock.cc:648] HybridClock initialized: now 1786402850148135 us; error 30 us; skew 500 ppm
Aug 10 23:00:50 dist-test-slave-6nmv krb5kdc[21724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402850, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:50.150836 21739 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:50.151729 21739 webserver.cc:460] Webserver started at http://127.29.78.190:41185/ using document root <none> and password file <none>
I20260810 23:00:50.152248 21739 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:50.152341 21739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:50.152578 21739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:50.154532 21739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "cbf66be906334a808b51582fc9c8d824"
format_stamp: "Formatted at 2026-08-10 23:00:50 on dist-test-slave-6nmv"
server_key: "f9fe925726e13656e535fcd697d84f17"
server_key_iv: "87f1e81dfcb2fc988375f835993baa2e"
server_key_version: "encryptionkey@0"
I20260810 23:00:50.155076 21739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "cbf66be906334a808b51582fc9c8d824"
format_stamp: "Formatted at 2026-08-10 23:00:50 on dist-test-slave-6nmv"
server_key: "f9fe925726e13656e535fcd697d84f17"
server_key_iv: "87f1e81dfcb2fc988375f835993baa2e"
server_key_version: "encryptionkey@0"
I20260810 23:00:50.158694 21739 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:00:50.160948 21757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:50.162118 21739 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:50.162259 21739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "cbf66be906334a808b51582fc9c8d824"
format_stamp: "Formatted at 2026-08-10 23:00:50 on dist-test-slave-6nmv"
server_key: "f9fe925726e13656e535fcd697d84f17"
server_key_iv: "87f1e81dfcb2fc988375f835993baa2e"
server_key_version: "encryptionkey@0"
I20260810 23:00:50.162365 21739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:50.182456 21739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:50.185086 21739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:50.185274 21739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:50.192631 21739 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:36713
I20260810 23:00:50.192647 21817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:36713 every 8 connection(s)
I20260810 23:00:50.193599 21739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:50.196177 21818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:50.201174 21818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824: Bootstrap starting.
I20260810 23:00:50.202222 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21739
I20260810 23:00:50.202373 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:50.202714 30010 external_mini_cluster.cc:1428] Setting key d3d4b87d0ccb1c7ccf1fd6fcbdf2653d
I20260810 23:00:50.203944 21818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:50.204630 21818 log.cc:824] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:50.206835 21818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824: No bootstrap required, opened a new log
Aug 10 23:00:50 dist-test-slave-6nmv krb5kdc[21724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:50.210949 21818 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf66be906334a808b51582fc9c8d824" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36713 } }
I20260810 23:00:50.211165 21818 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:50.211247 21818 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf66be906334a808b51582fc9c8d824, State: Initialized, Role: FOLLOWER
I20260810 23:00:50.211728 21818 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf66be906334a808b51582fc9c8d824" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36713 } }
I20260810 23:00:50.211863 21818 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:50.211951 21818 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:50.212085 21818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:50.212906 21818 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf66be906334a808b51582fc9c8d824" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36713 } }
I20260810 23:00:50.213254 21818 leader_election.cc:302] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf66be906334a808b51582fc9c8d824; no voters:
I20260810 23:00:50.213551 21818 leader_election.cc:288] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:50.213680 21823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:50.213933 21823 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [term 1 LEADER]: Becoming Leader. State: Replica: cbf66be906334a808b51582fc9c8d824, State: Running, Role: LEADER
I20260810 23:00:50.214315 21823 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf66be906334a808b51582fc9c8d824" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36713 } }
I20260810 23:00:50.214859 21818 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:50.216427 21824 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf66be906334a808b51582fc9c8d824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf66be906334a808b51582fc9c8d824" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36713 } } }
I20260810 23:00:50.216575 21824 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:50.216925 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf66be906334a808b51582fc9c8d824. Latest consensus state: current_term: 1 leader_uuid: "cbf66be906334a808b51582fc9c8d824" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf66be906334a808b51582fc9c8d824" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 36713 } } }
I20260810 23:00:50.217126 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:50.217082 21841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:50.218158 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:50.203682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51357 (local address 127.29.78.190:36713)
0810 23:00:50.204058 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:50.204093 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:50.204136 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:00:50.204682 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:50.204697 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:50.205111 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:50.205447 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:50.206409 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:50.207291 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:50.208404 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:50.208858 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:50.208970 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:50.211880 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:50.211888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:50.211909 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:50.211957 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:50.214093 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:50.214692 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:50.214697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:50.214714 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:50.214836 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:50.215219 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:50.215225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:50.215226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:50.215706 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:50.215917 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:50.217005 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0810 23:00:50.217234 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260810 23:00:50.220410 21841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:50.221174 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:50.226476 21841 catalog_manager.cc:1347] Generated new cluster ID: e4760d7d14f64f01a70fdf8a9e5e5cd2
I20260810 23:00:50.226583 21841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:50.231381 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:50.223445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41347 (local address 127.29.78.190:36713)
0810 23:00:50.223848 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:50.223852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:50.223865 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:00:50.223896 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:50.223898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:50.223940 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:50.224031 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:50.224789 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:50.225218 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:50.225988 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:50.226238 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:50.226328 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:50.227893 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:50.227897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:50.227900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:50.227922 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:50.229559 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:50.230119 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:50.230123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:50.230124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:50.230212 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:50.230531 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:50.230534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:50.230536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:50.230758 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:50.230921 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:50.231147 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 23:00:50.231239 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260810 23:00:50.243178 21841 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:50.244181 21841 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:50.252947 21841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf66be906334a808b51582fc9c8d824: Generated new TSK 0
I20260810 23:00:50.253521 21841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:50.264209 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21739
W20260810 23:00:50.271031 21722 connection.cc:535] client connection to 127.29.78.190:36713 recv error: Network error: failed to read from TLS socket (remote: 127.29.78.190:36713): Connection reset by peer (error 104)
2026-08-10T23:00:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:50 dist-test-slave-6nmv krb5kdc[21853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:50 dist-test-slave-6nmv krb5kdc[21853](info): set up 2 sockets
Aug 10 23:00:50 dist-test-slave-6nmv krb5kdc[21853](info): commencing operation
krb5kdc: starting...
W20260810 23:00:51.368436 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:51 dist-test-slave-6nmv krb5kdc[21853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:51.488245 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45089
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:51.592434 21871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:51.592674 21871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:51.592712 21871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:51.596663 21871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:51.596711 21871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:51.596730 21871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:51.596746 21871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:51.596761 21871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:51.601749 21871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45089
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:51.602592 21871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:51.603495 21871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:51.611624 21878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:51.611676 21880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:51.611676 21877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:51.611876 21871 server_base.cc:1034] running on GCE node
I20260810 23:00:51.612571 21871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:51.613636 21871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:51.614821 21871 hybrid_clock.cc:648] HybridClock initialized: now 1786402851614815 us; error 62 us; skew 500 ppm
Aug 10 23:00:51 dist-test-slave-6nmv krb5kdc[21853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402851, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:51.617408 21871 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:51.618232 21871 webserver.cc:460] Webserver started at http://127.29.78.190:46027/ using document root <none> and password file <none>
I20260810 23:00:51.618723 21871 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:51.618782 21871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:51.618947 21871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:51.620779 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "22edf46aad7f4f50958f683a9bc71ce0"
format_stamp: "Formatted at 2026-08-10 23:00:51 on dist-test-slave-6nmv"
server_key: "43b2c1984b9ce114e48cd109339bdc57"
server_key_iv: "22889fbedf55866321151f619b79d590"
server_key_version: "encryptionkey@0"
I20260810 23:00:51.621297 21871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "22edf46aad7f4f50958f683a9bc71ce0"
format_stamp: "Formatted at 2026-08-10 23:00:51 on dist-test-slave-6nmv"
server_key: "43b2c1984b9ce114e48cd109339bdc57"
server_key_iv: "22889fbedf55866321151f619b79d590"
server_key_version: "encryptionkey@0"
I20260810 23:00:51.624725 21871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 23:00:51.626844 21889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:51.628016 21871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:00:51.628168 21871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "22edf46aad7f4f50958f683a9bc71ce0"
format_stamp: "Formatted at 2026-08-10 23:00:51 on dist-test-slave-6nmv"
server_key: "43b2c1984b9ce114e48cd109339bdc57"
server_key_iv: "22889fbedf55866321151f619b79d590"
server_key_version: "encryptionkey@0"
I20260810 23:00:51.628255 21871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:51.656514 21871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:51.659390 21871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:51.659590 21871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:51.666389 21871 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45089
I20260810 23:00:51.666455 21948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45089 every 8 connection(s)
I20260810 23:00:51.667276 21871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:51.669792 21949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:51.674149 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21871
I20260810 23:00:51.674314 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:51.674611 30010 external_mini_cluster.cc:1428] Setting key 6998ebb261b6cb3ecea6fb2319b1f67d
I20260810 23:00:51.675458 21949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0: Bootstrap starting.
I20260810 23:00:51.677685 21949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:51.678416 21949 log.cc:824] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:51.680189 21949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0: No bootstrap required, opened a new log
I20260810 23:00:51.683172 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:51.675909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41901 (local address 127.29.78.190:45089)
0810 23:00:51.676317 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:51.676331 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:51.676361 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:00:51.676995 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:51.677009 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:51.677333 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:51.677655 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:51.678096 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:51.679103 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:51.680025 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:51.680539 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:51.680718 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:51.681565 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:51.681570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:51.681581 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:51.681999 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:51.682055 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:51.682078 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:51.682215 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:51.682433 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 23:00:51.682558 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
I20260810 23:00:51.683854 21949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22edf46aad7f4f50958f683a9bc71ce0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45089 } }
I20260810 23:00:51.684150 21949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:51.684238 21949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22edf46aad7f4f50958f683a9bc71ce0, State: Initialized, Role: FOLLOWER
I20260810 23:00:51.684659 21949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22edf46aad7f4f50958f683a9bc71ce0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45089 } }
I20260810 23:00:51.684823 21949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:51.684930 21949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:51.685065 21949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:51.686118 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:51.686107 21949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22edf46aad7f4f50958f683a9bc71ce0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45089 } }
I20260810 23:00:51.686424 21949 leader_election.cc:302] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22edf46aad7f4f50958f683a9bc71ce0; no voters:
I20260810 23:00:51.686734 21949 leader_election.cc:288] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:51.686792 21954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:51.686990 21954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 22edf46aad7f4f50958f683a9bc71ce0, State: Running, Role: LEADER
I20260810 23:00:51.687316 21954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22edf46aad7f4f50958f683a9bc71ce0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45089 } }
I20260810 23:00:51.687877 21949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:51.688441 21954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22edf46aad7f4f50958f683a9bc71ce0. Latest consensus state: current_term: 1 leader_uuid: "22edf46aad7f4f50958f683a9bc71ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22edf46aad7f4f50958f683a9bc71ce0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45089 } } }
I20260810 23:00:51.688651 21954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:51.688912 21955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22edf46aad7f4f50958f683a9bc71ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22edf46aad7f4f50958f683a9bc71ce0" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45089 } } }
I20260810 23:00:51.689024 21955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:51.689082 21968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:51.692111 21968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:51.696372 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:51.690199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57005 (local address 127.29.78.190:45089)
0810 23:00:51.690294 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:51.690297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:51.690659 (+ 362us) server_negotiation.cc:407] Connection header received
0810 23:00:51.690784 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:51.690786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:51.690822 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:51.690901 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:51.691201 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:51.691987 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:51.693192 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:51.693434 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:51.693539 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:51.695650 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:51.695654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:51.695656 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:51.695669 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:51.695695 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:51.695706 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:51.695765 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:51.696185 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 23:00:51.696216 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:00:51.697458 21968 catalog_manager.cc:1347] Generated new cluster ID: 2656322cd97048e9bd8331a407a98170
I20260810 23:00:51.697515 21968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:51.711042 21968 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:51.712152 21968 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:51.722522 21968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22edf46aad7f4f50958f683a9bc71ce0: Generated new TSK 0
I20260810 23:00:51.723132 21968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:51.730110 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21871
W20260810 23:00:51.739141 21851 connection.cc:535] client connection to 127.29.78.190:45089 recv error: Network error: failed to read from TLS socket (remote: 127.29.78.190:45089): Connection reset by peer (error 104)
2026-08-10T23:00:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1467 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:51 dist-test-slave-6nmv krb5kdc[21983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:51 dist-test-slave-6nmv krb5kdc[21983](info): set up 2 sockets
Aug 10 23:00:51 dist-test-slave-6nmv krb5kdc[21983](info): commencing operation
krb5kdc: starting...
W20260810 23:00:52.788158 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:52 dist-test-slave-6nmv krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:52.906262 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41217
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:34215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:53.012396 21996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:53.012686 21996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:53.012776 21996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:53.016348 21996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:53.016424 21996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:53.016459 21996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:53.016513 21996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:53.016551 21996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:53.021567 21996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:34215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:53.022580 21996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:53.023561 21996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:53.031569 22003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:53.031620 22002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:53.031670 22006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:53.031745 21996 server_base.cc:1034] running on GCE node
I20260810 23:00:53.032341 21996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:53.033329 21996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:53.034484 21996 hybrid_clock.cc:648] HybridClock initialized: now 1786402853034468 us; error 27 us; skew 500 ppm
Aug 10 23:00:53 dist-test-slave-6nmv krb5kdc[21983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402853, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:53.045986 21996 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:53.046825 21996 webserver.cc:460] Webserver started at http://127.29.78.190:38727/ using document root <none> and password file <none>
I20260810 23:00:53.047320 21996 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:53.047375 21996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:53.047542 21996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:53.049642 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "29fd0e86a673471f91c58b891f0289ed"
format_stamp: "Formatted at 2026-08-10 23:00:53 on dist-test-slave-6nmv"
server_key: "ec7afac5b7fb75e9af857e2062952e75"
server_key_iv: "a14c9a78556abb30fbec675ce47d56a7"
server_key_version: "encryptionkey@0"
I20260810 23:00:53.050246 21996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "29fd0e86a673471f91c58b891f0289ed"
format_stamp: "Formatted at 2026-08-10 23:00:53 on dist-test-slave-6nmv"
server_key: "ec7afac5b7fb75e9af857e2062952e75"
server_key_iv: "a14c9a78556abb30fbec675ce47d56a7"
server_key_version: "encryptionkey@0"
I20260810 23:00:53.053783 21996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:00:53.056044 22013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:53.057349 21996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:53.057512 21996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "29fd0e86a673471f91c58b891f0289ed"
format_stamp: "Formatted at 2026-08-10 23:00:53 on dist-test-slave-6nmv"
server_key: "ec7afac5b7fb75e9af857e2062952e75"
server_key_iv: "a14c9a78556abb30fbec675ce47d56a7"
server_key_version: "encryptionkey@0"
I20260810 23:00:53.057621 21996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:53.061924 21996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:53.064951 21996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:53.065136 21996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:53.072026 21996 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41217
I20260810 23:00:53.072052 22070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41217 every 8 connection(s)
I20260810 23:00:53.072873 21996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:53.075449 22071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:53.080565 22071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed: Bootstrap starting.
I20260810 23:00:53.082160 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 21996
I20260810 23:00:53.082320 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:53.082581 30010 external_mini_cluster.cc:1428] Setting key c650d0ef9dd15fc385af540a48bf045f
I20260810 23:00:53.083225 22071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:53.083956 22071 log.cc:824] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:53.086334 22071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed: No bootstrap required, opened a new log
I20260810 23:00:53.087617 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:53.083580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59945 (local address 127.29.78.190:41217)
0810 23:00:53.083999 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:53.084012 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:53.084044 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:00:53.084890 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:53.084908 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:53.085166 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:53.085423 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:53.086289 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:53.086293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:53.086303 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:53.086312 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:53.086394 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:53.086412 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:53.086588 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:53.086722 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 23:00:53.086875 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20260810 23:00:53.090397 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:53.090655 22071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd0e86a673471f91c58b891f0289ed" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41217 } }
I20260810 23:00:53.090903 22071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:53.090987 22071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29fd0e86a673471f91c58b891f0289ed, State: Initialized, Role: FOLLOWER
I20260810 23:00:53.091452 22071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd0e86a673471f91c58b891f0289ed" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41217 } }
I20260810 23:00:53.091637 22071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:53.091713 22071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:53.091830 22071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:53.092999 22071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd0e86a673471f91c58b891f0289ed" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41217 } }
I20260810 23:00:53.093357 22071 leader_election.cc:302] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29fd0e86a673471f91c58b891f0289ed; no voters:
I20260810 23:00:53.093683 22071 leader_election.cc:288] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:53.093814 22082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:53.094084 22082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [term 1 LEADER]: Becoming Leader. State: Replica: 29fd0e86a673471f91c58b891f0289ed, State: Running, Role: LEADER
I20260810 23:00:53.094409 22082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd0e86a673471f91c58b891f0289ed" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41217 } }
I20260810 23:00:53.094539 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:53.092722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50115 (local address 127.29.78.190:41217)
0810 23:00:53.092852 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:53.092856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:53.093041 (+ 185us) server_negotiation.cc:407] Connection header received
0810 23:00:53.093191 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:53.093194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:53.093224 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:53.093295 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:53.094074 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:53.094077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:53.094080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:53.094090 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:53.094115 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:53.094126 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:53.094214 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:53.094361 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 23:00:53.094398 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:00:53.094977 22071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:53.095621 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29fd0e86a673471f91c58b891f0289ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd0e86a673471f91c58b891f0289ed" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41217 } } }
I20260810 23:00:53.095734 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:53.095949 22082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29fd0e86a673471f91c58b891f0289ed. Latest consensus state: current_term: 1 leader_uuid: "29fd0e86a673471f91c58b891f0289ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29fd0e86a673471f91c58b891f0289ed" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41217 } } }
I20260810 23:00:53.096079 22091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:53.096095 22082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:53.099205 22091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:53.104632 22091 catalog_manager.cc:1347] Generated new cluster ID: c8c6d807173a4fe2a16cb8dee04b35b3
I20260810 23:00:53.104703 22091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:53.127789 22091 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:53.128050 22091 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:53.132578 22091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29fd0e86a673471f91c58b891f0289ed: Generated new TSK 0
I20260810 23:00:53.133136 22091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:53.138273 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 21996
2026-08-10T23:00:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:53 dist-test-slave-6nmv krb5kdc[22104](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:53 dist-test-slave-6nmv krb5kdc[22104](info): set up 2 sockets
Aug 10 23:00:53 dist-test-slave-6nmv krb5kdc[22104](info): commencing operation
krb5kdc: starting...
W20260810 23:00:54.236487 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:54 dist-test-slave-6nmv krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:54.352352 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41437
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:37167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:41437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:54.457795 22118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:54.458029 22118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:54.458066 22118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:54.461730 22118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:54.461772 22118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:54.461789 22118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:54.461803 22118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:54.461885 22118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:54.466540 22118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:37167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:41437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:41437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:54.467420 22118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:54.468287 22118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:00:54.476507 22118 server_base.cc:1034] running on GCE node
W20260810 23:00:54.476442 22126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:54.476442 22123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:54.476426 22124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:54.477281 22118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:54.478392 22118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:54.479604 22118 hybrid_clock.cc:648] HybridClock initialized: now 1786402854479586 us; error 29 us; skew 500 ppm
Aug 10 23:00:54 dist-test-slave-6nmv krb5kdc[22104](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402854, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:54.482149 22118 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:54.482941 22118 webserver.cc:460] Webserver started at http://127.29.78.190:46639/ using document root <none> and password file <none>
I20260810 23:00:54.483431 22118 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:54.483523 22118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:54.483737 22118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:54.485482 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "022d63cad156419488c30f6686f7ea83"
format_stamp: "Formatted at 2026-08-10 23:00:54 on dist-test-slave-6nmv"
server_key: "85c97348e70f6f8febc31b88119e22ab"
server_key_iv: "8a67562da8fbf97dc1ff8428e70ffa5b"
server_key_version: "encryptionkey@0"
I20260810 23:00:54.486042 22118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "022d63cad156419488c30f6686f7ea83"
format_stamp: "Formatted at 2026-08-10 23:00:54 on dist-test-slave-6nmv"
server_key: "85c97348e70f6f8febc31b88119e22ab"
server_key_iv: "8a67562da8fbf97dc1ff8428e70ffa5b"
server_key_version: "encryptionkey@0"
I20260810 23:00:54.489470 22118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 23:00:54.491863 22133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:54.492960 22118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:54.493098 22118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "022d63cad156419488c30f6686f7ea83"
format_stamp: "Formatted at 2026-08-10 23:00:54 on dist-test-slave-6nmv"
server_key: "85c97348e70f6f8febc31b88119e22ab"
server_key_iv: "8a67562da8fbf97dc1ff8428e70ffa5b"
server_key_version: "encryptionkey@0"
I20260810 23:00:54.493203 22118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:54.516175 22118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:54.518864 22118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:54.519068 22118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:54.525887 22118 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:41437
I20260810 23:00:54.525945 22192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:41437 every 8 connection(s)
I20260810 23:00:54.526801 22118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:54.528190 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 22118
I20260810 23:00:54.528334 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:54.528640 30010 external_mini_cluster.cc:1428] Setting key afe35962cd2545a5c1e931a23bb40881
I20260810 23:00:54.529856 22193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:54.535809 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83: Bootstrap starting.
Aug 10 23:00:54 dist-test-slave-6nmv krb5kdc[22104](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:54.538321 22193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:54.539243 22193 log.cc:824] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:54.541414 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83: No bootstrap required, opened a new log
I20260810 23:00:54.542426 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:54.530015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55299 (local address 127.29.78.190:41437)
0810 23:00:54.530398 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:54.530412 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:54.530445 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:00:54.531007 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:54.531021 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:54.531303 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:54.531551 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:54.532432 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:54.533379 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:54.534480 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:54.534886 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:54.534991 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:54.537632 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:54.537639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:54.537652 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:54.537681 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:54.539563 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:54.540087 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:54.540092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:54.540097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:54.540223 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:54.540496 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:54.540499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:54.540500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:54.540954 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:54.541140 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:54.541333 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:00:54.541598 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":127,"threads_started":1}
I20260810 23:00:54.544931 22193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d63cad156419488c30f6686f7ea83" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41437 } }
I20260810 23:00:54.545166 22193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:54.545239 22193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 022d63cad156419488c30f6686f7ea83, State: Initialized, Role: FOLLOWER
I20260810 23:00:54.545490 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:54.545673 22193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d63cad156419488c30f6686f7ea83" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41437 } }
I20260810 23:00:54.545830 22193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:54.545890 22193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:54.545994 22193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:54.547034 22193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d63cad156419488c30f6686f7ea83" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41437 } }
I20260810 23:00:54.547351 22193 leader_election.cc:302] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 022d63cad156419488c30f6686f7ea83; no voters:
I20260810 23:00:54.547678 22193 leader_election.cc:288] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:54.547827 22203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:54.548000 22203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [term 1 LEADER]: Becoming Leader. State: Replica: 022d63cad156419488c30f6686f7ea83, State: Running, Role: LEADER
I20260810 23:00:54.548305 22203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d63cad156419488c30f6686f7ea83" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41437 } }
I20260810 23:00:54.548780 22193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:54.550024 22205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "022d63cad156419488c30f6686f7ea83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d63cad156419488c30f6686f7ea83" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41437 } } }
I20260810 23:00:54.550374 22205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:54.550276 22206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 022d63cad156419488c30f6686f7ea83. Latest consensus state: current_term: 1 leader_uuid: "022d63cad156419488c30f6686f7ea83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "022d63cad156419488c30f6686f7ea83" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 41437 } } }
I20260810 23:00:54.551136 22206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:54.551493 22216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:54.554684 22216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:54.558302 22196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:54.547647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41579 (local address 127.29.78.190:41437)
0810 23:00:54.547744 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:54.547748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:54.550289 (+ 2541us) server_negotiation.cc:407] Connection header received
0810 23:00:54.550509 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:54.550513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:54.550561 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:54.550614 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:54.551563 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:54.552327 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:54.553380 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:54.553692 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:54.553762 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:54.555273 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:54.555277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:54.555279 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:54.555300 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:54.556763 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:54.557156 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:54.557159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:54.557161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:54.557234 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:54.557495 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:54.557498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:54.557499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:54.557648 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:54.557777 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:54.558014 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 23:00:54.558136 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 23:00:54.560192 22216 catalog_manager.cc:1347] Generated new cluster ID: 5c6285197d80422f978f820fe6af51da
I20260810 23:00:54.560258 22216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:54.570725 22216 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:54.571457 22216 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:54.576069 22216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 022d63cad156419488c30f6686f7ea83: Generated new TSK 0
I20260810 23:00:54.576596 22216 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:54.580781 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 22118
2026-08-10T23:00:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:54 dist-test-slave-6nmv krb5kdc[22227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:54 dist-test-slave-6nmv krb5kdc[22227](info): set up 2 sockets
Aug 10 23:00:54 dist-test-slave-6nmv krb5kdc[22227](info): commencing operation
krb5kdc: starting...
W20260810 23:00:55.631956 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:55 dist-test-slave-6nmv krb5kdc[22227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:55.747128 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45775
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:44513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:45775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:55.854981 22241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:55.855194 22241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:55.855230 22241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:55.858814 22241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:55.858857 22241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:55.858894 22241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:55.858911 22241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:55.858979 22241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:55.863947 22241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:44513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:45775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:45775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:55.864835 22241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:55.865727 22241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:55.873477 22246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:55.873601 22241 server_base.cc:1034] running on GCE node
W20260810 23:00:55.873507 22247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:55.873486 22249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:55.874295 22241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:55.875311 22241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:55.876492 22241 hybrid_clock.cc:648] HybridClock initialized: now 1786402855876475 us; error 29 us; skew 500 ppm
Aug 10 23:00:55 dist-test-slave-6nmv krb5kdc[22227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402855, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:55.879227 22241 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:55.880093 22241 webserver.cc:460] Webserver started at http://127.29.78.190:39533/ using document root <none> and password file <none>
I20260810 23:00:55.880599 22241 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:55.880684 22241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:55.880890 22241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:55.882822 22241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "23ad0425f2f04d0ea2b0365257b7849f"
format_stamp: "Formatted at 2026-08-10 23:00:55 on dist-test-slave-6nmv"
server_key: "867675b97815ecfa297d3b8339d56f40"
server_key_iv: "0904af6da2bc134455151e780450fe6d"
server_key_version: "encryptionkey@0"
I20260810 23:00:55.883347 22241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "23ad0425f2f04d0ea2b0365257b7849f"
format_stamp: "Formatted at 2026-08-10 23:00:55 on dist-test-slave-6nmv"
server_key: "867675b97815ecfa297d3b8339d56f40"
server_key_iv: "0904af6da2bc134455151e780450fe6d"
server_key_version: "encryptionkey@0"
I20260810 23:00:55.886641 22241 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 23:00:55.888772 22256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:55.889990 22241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:00:55.890118 22241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "23ad0425f2f04d0ea2b0365257b7849f"
format_stamp: "Formatted at 2026-08-10 23:00:55 on dist-test-slave-6nmv"
server_key: "867675b97815ecfa297d3b8339d56f40"
server_key_iv: "0904af6da2bc134455151e780450fe6d"
server_key_version: "encryptionkey@0"
I20260810 23:00:55.890232 22241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:55.919145 22241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:55.921769 22241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:55.921985 22241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:55.928560 22241 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:45775
I20260810 23:00:55.928578 22311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:45775 every 8 connection(s)
I20260810 23:00:55.929428 22241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:55.932017 22312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:55.933033 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 22241
I20260810 23:00:55.933187 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:55.933506 30010 external_mini_cluster.cc:1428] Setting key ac5c5f93523fc6d0035711a913ff456a
I20260810 23:00:55.938149 22312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f: Bootstrap starting.
I20260810 23:00:55.940889 22312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:55.941637 22312 log.cc:824] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:55.942972 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:55.934797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58295 (local address 127.29.78.190:45775)
0810 23:00:55.935172 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:55.935187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:55.935220 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:00:55.935820 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:55.935834 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:55.936146 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:55.936398 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:55.936771 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:55.937677 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:55.938848 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:55.939573 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:55.939724 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:55.940812 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:55.940821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:55.940838 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:55.941356 (+ 518us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:55.941433 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:55.941469 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:55.941598 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:55.941768 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 23:00:55.942027 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":118,"threads_started":1}
I20260810 23:00:55.943953 22312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f: No bootstrap required, opened a new log
I20260810 23:00:55.946162 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:55.949368 22312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad0425f2f04d0ea2b0365257b7849f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45775 } }
I20260810 23:00:55.949667 22312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:55.949733 22312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23ad0425f2f04d0ea2b0365257b7849f, State: Initialized, Role: FOLLOWER
I20260810 23:00:55.950205 22312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad0425f2f04d0ea2b0365257b7849f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45775 } }
I20260810 23:00:55.950379 22312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:55.950453 22312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:55.950549 22312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:55.951639 22312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad0425f2f04d0ea2b0365257b7849f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45775 } }
I20260810 23:00:55.952015 22312 leader_election.cc:302] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23ad0425f2f04d0ea2b0365257b7849f; no voters:
I20260810 23:00:55.952400 22312 leader_election.cc:288] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:55.952466 22324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:55.952670 22324 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [term 1 LEADER]: Becoming Leader. State: Replica: 23ad0425f2f04d0ea2b0365257b7849f, State: Running, Role: LEADER
I20260810 23:00:55.952986 22316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:55.948600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48569 (local address 127.29.78.190:45775)
0810 23:00:55.948698 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:55.948701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:55.948952 (+ 251us) server_negotiation.cc:407] Connection header received
0810 23:00:55.949023 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:55.949026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:55.949071 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:55.949153 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:55.949437 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:55.950228 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:55.951149 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:55.951528 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:55.951656 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:55.952472 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:55.952478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:55.952483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:55.952496 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:55.952526 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:55.952542 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:55.952615 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:55.952808 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:00:55.952855 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:00:55.952952 22324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad0425f2f04d0ea2b0365257b7849f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45775 } }
I20260810 23:00:55.953765 22312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:55.955029 22325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23ad0425f2f04d0ea2b0365257b7849f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad0425f2f04d0ea2b0365257b7849f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45775 } } }
I20260810 23:00:55.955070 22326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23ad0425f2f04d0ea2b0365257b7849f. Latest consensus state: current_term: 1 leader_uuid: "23ad0425f2f04d0ea2b0365257b7849f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23ad0425f2f04d0ea2b0365257b7849f" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 45775 } } }
I20260810 23:00:55.955147 22325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:55.955170 22326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:55.955617 22334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:55.958653 22334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:55.963680 22334 catalog_manager.cc:1347] Generated new cluster ID: a85ea738c5a04acca24ebfba6363019c
I20260810 23:00:55.963738 22334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:55.980450 22334 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:55.981698 22334 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:55.990744 22334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23ad0425f2f04d0ea2b0365257b7849f: Generated new TSK 0
I20260810 23:00:55.991492 22334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:55.996014 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 22241
2026-08-10T23:00:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:56 dist-test-slave-6nmv krb5kdc[22349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:56 dist-test-slave-6nmv krb5kdc[22349](info): set up 2 sockets
Aug 10 23:00:56 dist-test-slave-6nmv krb5kdc[22349](info): commencing operation
krb5kdc: starting...
W20260810 23:00:57.058472 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:57 dist-test-slave-6nmv krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:57.178985 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40487
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:40701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:40487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:57.285519 22365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:57.285811 22365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:57.285920 22365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:57.289848 22365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:57.289935 22365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:57.289974 22365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:57.290032 22365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:57.290072 22365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:57.295136 22365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:40701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:40487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:57.296113 22365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:57.297173 22365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:57.305531 22376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:57.305524 22374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:57.305524 22373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:57.305773 22365 server_base.cc:1034] running on GCE node
I20260810 23:00:57.306440 22365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:57.307438 22365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:57.308626 22365 hybrid_clock.cc:648] HybridClock initialized: now 1786402857308603 us; error 35 us; skew 500 ppm
Aug 10 23:00:57 dist-test-slave-6nmv krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402857, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:57.311282 22365 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:57.312157 22365 webserver.cc:460] Webserver started at http://127.29.78.190:45247/ using document root <none> and password file <none>
I20260810 23:00:57.312687 22365 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:57.312778 22365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:57.313016 22365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:57.315009 22365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "c9fa3b89449f4d8a86b8c828a2216c64"
format_stamp: "Formatted at 2026-08-10 23:00:57 on dist-test-slave-6nmv"
server_key: "3051aff445d61af167f32ede4de8f39c"
server_key_iv: "fa280b8d7f5d0a0e9d27345cccfd9475"
server_key_version: "encryptionkey@0"
I20260810 23:00:57.315558 22365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "c9fa3b89449f4d8a86b8c828a2216c64"
format_stamp: "Formatted at 2026-08-10 23:00:57 on dist-test-slave-6nmv"
server_key: "3051aff445d61af167f32ede4de8f39c"
server_key_iv: "fa280b8d7f5d0a0e9d27345cccfd9475"
server_key_version: "encryptionkey@0"
I20260810 23:00:57.319093 22365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 23:00:57.321328 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:57.322499 22365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:00:57.322628 22365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "c9fa3b89449f4d8a86b8c828a2216c64"
format_stamp: "Formatted at 2026-08-10 23:00:57 on dist-test-slave-6nmv"
server_key: "3051aff445d61af167f32ede4de8f39c"
server_key_iv: "fa280b8d7f5d0a0e9d27345cccfd9475"
server_key_version: "encryptionkey@0"
I20260810 23:00:57.322728 22365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:57.326893 22365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:57.329439 22365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:57.329638 22365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:57.336719 22365 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:40487
I20260810 23:00:57.336745 22441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:40487 every 8 connection(s)
I20260810 23:00:57.337607 22365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:57.340340 22442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:57.344794 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 22365
I20260810 23:00:57.344959 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:57.345237 30010 external_mini_cluster.cc:1428] Setting key 1a7b85de6ffc30db4dd904f467c2d9b6
I20260810 23:00:57.346077 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64: Bootstrap starting.
I20260810 23:00:57.348699 22442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:57.349686 22442 log.cc:824] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:57.351177 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:57.346710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58737 (local address 127.29.78.190:40487)
0810 23:00:57.347160 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:57.347184 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:57.347230 (+ 46us) server_negotiation.cc:407] Connection header received
0810 23:00:57.348038 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:57.348061 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:57.348393 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:57.348767 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:57.349675 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:57.349680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:57.349697 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:57.349714 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:57.349780 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:57.349830 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:57.349947 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:57.350060 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 23:00:57.350322 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":120,"threads_started":1}
I20260810 23:00:57.351879 22442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64: No bootstrap required, opened a new log
I20260810 23:00:57.353911 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:57.356289 22442 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40487 } }
I20260810 23:00:57.356575 22442 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:00:57.356637 22442 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fa3b89449f4d8a86b8c828a2216c64, State: Initialized, Role: FOLLOWER
I20260810 23:00:57.357100 22442 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40487 } }
I20260810 23:00:57.357260 22442 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:00:57.357339 22442 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:00:57.357453 22442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:57.357844 22445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:57.356198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38925 (local address 127.29.78.190:40487)
0810 23:00:57.356313 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:57.356316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:57.356448 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:00:57.356577 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:57.356582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:57.356621 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:57.356719 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:57.357357 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:57.357361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:57.357363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:00:57.357376 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:57.357404 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:00:57.357417 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:57.357479 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:57.357650 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:00:57.357684 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:00:57.358490 22442 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40487 } }
I20260810 23:00:57.358829 22442 leader_election.cc:302] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9fa3b89449f4d8a86b8c828a2216c64; no voters:
I20260810 23:00:57.359197 22442 leader_election.cc:288] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:00:57.359321 22453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:00:57.359509 22453 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [term 1 LEADER]: Becoming Leader. State: Replica: c9fa3b89449f4d8a86b8c828a2216c64, State: Running, Role: LEADER
I20260810 23:00:57.359793 22453 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40487 } }
I20260810 23:00:57.360447 22442 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:57.361428 22454 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40487 } } }
I20260810 23:00:57.361573 22454 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:57.361907 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9fa3b89449f4d8a86b8c828a2216c64. Latest consensus state: current_term: 1 leader_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa3b89449f4d8a86b8c828a2216c64" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 40487 } } }
I20260810 23:00:57.362010 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:57.362114 22465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:57.365255 22465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:57.370656 22465 catalog_manager.cc:1347] Generated new cluster ID: 39b1d68d8b1e4764bd4acd4cb7fb695c
I20260810 23:00:57.370718 22465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:57.395738 22465 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:57.396174 22465 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:57.401485 22465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9fa3b89449f4d8a86b8c828a2216c64: Generated new TSK 0
I20260810 23:00:57.402220 22465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:00:57.408839 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 22365
2026-08-10T23:00:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1413 ms)
[----------] 18 tests from AuthTokenIssuingTest (25468 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:00:57 dist-test-slave-6nmv krb5kdc[22478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:00:57 dist-test-slave-6nmv krb5kdc[22478](info): set up 2 sockets
Aug 10 23:00:57 dist-test-slave-6nmv krb5kdc[22478](info): commencing operation
krb5kdc: starting...
W20260810 23:00:58.451843 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:00:58 dist-test-slave-6nmv krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:00:58.570899 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43725
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43725,127.29.78.189:33389,127.29.78.188:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:58.675704 22492 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:00:58.675951 22492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:58.676003 22492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:58.676075 22492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:58.679955 22492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:58.680002 22492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:58.680022 22492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:58.680074 22492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:58.680106 22492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:58.685001 22492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43725,127.29.78.189:33389,127.29.78.188:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:43725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:58.685992 22492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:58.686890 22492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:58.695142 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:58.695251 22492 server_base.cc:1034] running on GCE node
W20260810 23:00:58.695188 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:58.695174 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:58.695823 22492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:58.696758 22492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:58.697934 22492 hybrid_clock.cc:648] HybridClock initialized: now 1786402858697913 us; error 34 us; skew 500 ppm
Aug 10 23:00:58 dist-test-slave-6nmv krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:58.704087 22492 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:00:58.704867 22492 webserver.cc:460] Webserver started at http://127.29.78.190:46227/ using document root <none> and password file <none>
I20260810 23:00:58.705369 22492 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:58.705452 22492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:58.705683 22492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:58.707592 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91"
format_stamp: "Formatted at 2026-08-10 23:00:58 on dist-test-slave-6nmv"
server_key: "b5cd8c9269c7fdf01e90dcfd3397d6b2"
server_key_iv: "7ed0ac3eed461714b207bc8ff49f0e8e"
server_key_version: "encryptionkey@0"
I20260810 23:00:58.708124 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91"
format_stamp: "Formatted at 2026-08-10 23:00:58 on dist-test-slave-6nmv"
server_key: "b5cd8c9269c7fdf01e90dcfd3397d6b2"
server_key_iv: "7ed0ac3eed461714b207bc8ff49f0e8e"
server_key_version: "encryptionkey@0"
I20260810 23:00:58.711578 22492 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 23:00:58.713795 22507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:58.714954 22492 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:58.715093 22492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91"
format_stamp: "Formatted at 2026-08-10 23:00:58 on dist-test-slave-6nmv"
server_key: "b5cd8c9269c7fdf01e90dcfd3397d6b2"
server_key_iv: "7ed0ac3eed461714b207bc8ff49f0e8e"
server_key_version: "encryptionkey@0"
I20260810 23:00:58.715201 22492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:58.739365 22492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:58.742409 22492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:58.742709 22492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:58.749725 22492 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:43725
I20260810 23:00:58.750030 22566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:43725 every 8 connection(s)
I20260810 23:00:58.750723 22492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:00:58.753154 22567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:58.756669 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 22492
I20260810 23:00:58.756366 22567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:00:58.756815 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:00:58.757097 30010 external_mini_cluster.cc:1428] Setting key 9fe7a6b843edd7da34baf6d719bdfc98
WARNING: no policy specified for kudu/127.29.78.189@KRBTEST.COM; defaulting to no policy
Aug 10 23:00:58 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.189@KRBTEST.COM" created.
I20260810 23:00:58.771214 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.759496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57926 (local address 127.29.78.190:43725)
0810 23:00:58.759795 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:58.759812 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:58.759856 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:00:58.760341 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:58.760347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:58.760542 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:58.760678 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:58.761753 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.762651 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.764216 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.764555 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.764661 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.766920 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:58.766924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:58.766933 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:58.766951 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:58.768316 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.768821 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.768826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.768830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.768935 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.769303 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.769308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.769312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.769788 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:58.769950 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:58.770301 (+ 351us) server_negotiation.cc:299] Negotiation successful
0810 23:00:58.770532 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260810 23:00:58.771220 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.758673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43725 (local address 127.0.0.1:57926)
0810 23:00:58.759094 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.759141 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:58.760051 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:58.760734 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:58.760742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:58.761174 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:58.761574 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.761598 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.762785 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.762792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.764024 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.764038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.764713 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.764720 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.764833 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.765299 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:58.765322 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:58.766801 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:58.768403 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.768411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.768416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.768712 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.769040 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.769046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.769051 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.769190 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.769929 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:58.769933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:58.770054 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 23:00:58.770187 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 23:00:58.770421 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":135,"threads_started":1}
I20260810 23:00:58.773375 22567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:00:58.774034 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.773834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49762)
0810 23:00:58.773925 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.773990 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 23:00:58.774224 22509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
I20260810 23:00:58.774997 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.774808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49772)
0810 23:00:58.774904 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.774949 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:00:58.775141 22567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:33389: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
I20260810 23:00:58.784345 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:33389
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43725,127.29.78.189:33389,127.29.78.188:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:00:58.824417 22567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } attempt: 1
I20260810 23:00:58.825186 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.824927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49786)
0810 23:00:58.825068 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.825121 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:00:58.825924 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.825664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49796)
0810 23:00:58.825780 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.825866 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:00:58.826046 22567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:33389: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
W20260810 23:00:58.892122 22574 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:00:58.892457 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:58.892549 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:58.892639 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:00:58.894181 22567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } attempt: 2
I20260810 23:00:58.894953 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.894678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49804)
0810 23:00:58.894817 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.894882 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 23:00:58.895689 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.895465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49812)
0810 23:00:58.895577 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.895625 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:00:58.895797 22567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:33389: Network error: Client connection negotiation failed: client connection to 127.29.78.189:33389: connect: Connection refused (error 111)
W20260810 23:00:58.897312 22574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:58.897394 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:58.897451 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:58.897516 22574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:58.897560 22574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:58.903278 22574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43725,127.29.78.189:33389,127.29.78.188:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:58.904282 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:58.905252 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:00:58.913797 22583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:58.913830 22581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:58.914134 22580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:58.914126 22574 server_base.cc:1034] running on GCE node
I20260810 23:00:58.914625 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:58.915565 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:58.916752 22574 hybrid_clock.cc:648] HybridClock initialized: now 1786402858916739 us; error 21 us; skew 500 ppm
Aug 10 23:00:58 dist-test-slave-6nmv krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:58.919701 22574 init.cc:377] Logged in from keytab as kudu/127.29.78.189@KRBTEST.COM (short username kudu)
I20260810 23:00:58.920585 22574 webserver.cc:460] Webserver started at http://127.29.78.189:46425/ using document root <none> and password file <none>
I20260810 23:00:58.921097 22574 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:58.921157 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:58.921394 22574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:58.923246 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data/instance:
uuid: "d0f821c400604a25a482d8993caee706"
format_stamp: "Formatted at 2026-08-10 23:00:58 on dist-test-slave-6nmv"
server_key: "95cb569b4581d9894d66b1ba0bbdae70"
server_key_iv: "61b031e630ca2b40f51661d4771d0e07"
server_key_version: "encryptionkey@0"
I20260810 23:00:58.923780 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/wal/instance:
uuid: "d0f821c400604a25a482d8993caee706"
format_stamp: "Formatted at 2026-08-10 23:00:58 on dist-test-slave-6nmv"
server_key: "95cb569b4581d9894d66b1ba0bbdae70"
server_key_iv: "61b031e630ca2b40f51661d4771d0e07"
server_key_version: "encryptionkey@0"
I20260810 23:00:58.927127 22574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:00:58.929199 22590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:58.930375 22574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:58.930560 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/wal
uuid: "d0f821c400604a25a482d8993caee706"
format_stamp: "Formatted at 2026-08-10 23:00:58 on dist-test-slave-6nmv"
server_key: "95cb569b4581d9894d66b1ba0bbdae70"
server_key_iv: "61b031e630ca2b40f51661d4771d0e07"
server_key_version: "encryptionkey@0"
I20260810 23:00:58.930646 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:58.952203 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:58.954779 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:58.954965 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:58.961488 22574 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:33389
I20260810 23:00:58.961796 22645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:33389 every 8 connection(s)
I20260810 23:00:58.962498 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
I20260810 23:00:58.965004 22646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:58.968498 22646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:00:58.970193 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 22574
I20260810 23:00:58.970309 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-1/wal/instance
I20260810 23:00:58.970583 30010 external_mini_cluster.cc:1428] Setting key bfe17cb16fabf3a3674c9b902197845a
Aug 10 23:00:58 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.29.78.188@KRBTEST.COM; defaulting to no policy
I20260810 23:00:58.981165 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.970212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.29.78.190:43725)
0810 23:00:58.970348 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:58.970369 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:58.971205 (+ 836us) server_negotiation.cc:407] Connection header received
0810 23:00:58.972063 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:58.972066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:58.972131 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:58.972230 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:58.973266 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.973764 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.974892 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.975134 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.975246 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.977503 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:58.977507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:58.977510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:58.977541 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:58.979070 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.979665 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.979668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.979669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.979750 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.980056 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.980058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.980059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.980229 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:58.980306 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:58.980840 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 23:00:58.980941 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.188@KRBTEST.COM" created.
I20260810 23:00:58.981980 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.970653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43725 (local address 127.0.0.1:57928)
0810 23:00:58.971076 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.971111 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:58.971859 (+ 748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:58.972383 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:58.972389 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:58.972702 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:58.973118 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.973129 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.973925 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.973928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.974765 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.974773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.975282 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.975286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.975443 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.975998 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:58.976013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:58.977338 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:58.979228 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.979233 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.979245 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.979560 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.979841 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.979844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.979845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.979972 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.980317 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:58.980321 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:58.980633 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 23:00:58.980777 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 23:00:58.980987 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":128,"threads_started":1}
I20260810 23:00:58.983363 22646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:00:58.986923 22567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } attempt: 3
Aug 10 23:00:58 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 10 23:00:58 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260810 23:00:58.993148 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.983818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49820)
0810 23:00:58.983925 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.983940 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:58.984018 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:58.984582 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:58.984585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:58.984743 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:58.984824 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.984829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.985787 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.985793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.986338 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.986345 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.986863 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.986865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.987109 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.987673 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:58.987693 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:58.989001 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:58.991417 (+ 2416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.991420 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.991422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.991714 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.992086 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.992091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.992093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.992176 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.992663 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:58.992666 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:58.992779 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 23:00:58.992949 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 23:00:58.993047 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":88}
I20260810 23:00:58.993577 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.983964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.29.78.189:33389)
0810 23:00:58.984261 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:58.984269 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:58.984303 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:00:58.984332 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:58.984336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:58.984461 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:58.984601 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:58.984926 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.985675 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.986445 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.986758 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.986866 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.989126 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:58.989134 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:58.989142 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:58.989160 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:58.991296 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.991908 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.991913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.991914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.991993 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.992270 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.992275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.992281 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.992562 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:58.992702 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:58.993056 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 23:00:58.993196 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":226,"thread_start_us":129,"threads_started":1}
I20260810 23:00:58.995435 22646 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:00:58.996215 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.995988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43632)
0810 23:00:58.996086 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.996154 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 23:00:58.996431 22591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
I20260810 23:00:58.997349 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.987593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49828)
0810 23:00:58.987705 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.987719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:58.987790 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:58.988652 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:58.988657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:58.988830 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:58.988950 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.988956 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.989891 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.989894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.991025 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:58.991035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.991504 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.991506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:58.991599 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.992095 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:58.992107 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:58.993614 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:58.995601 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.995603 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.995604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.995859 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.996191 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:58.996195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:58.996197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:58.996279 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:58.996892 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:58.996895 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:58.997013 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 23:00:58.997103 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:00:58.997233 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:00:58.997357 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.997111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43648)
0810 23:00:58.997232 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.997287 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 23:00:58.997488 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:33419
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:45131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:43725,127.29.78.189:33389,127.29.78.188:33419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:00:58.997859 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33419: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
I20260810 23:00:58.998059 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.987367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.29.78.189:33389)
0810 23:00:58.988374 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:58.988385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:58.988402 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:00:58.988437 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:58.988441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:58.988502 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:58.988595 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:58.989082 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.989705 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.991144 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:58.991405 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:58.991515 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:58.993787 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:58.993790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:58.993792 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:58.993866 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:58.995447 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.995996 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.995999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.996000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.996091 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:58.996414 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:58.996419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:58.996423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:58.996772 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:58.997632 (+ 860us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:58.997704 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 23:00:58.997812 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":936,"thread_start_us":97,"threads_started":1}
I20260810 23:00:58.998638 22567 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:00:58.999310 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.999075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43656)
0810 23:00:58.999185 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.999242 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:00:58.999927 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:58.999705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43670)
0810 23:00:58.999846 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:58.999885 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260810 23:00:59.000092 22567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33419: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
I20260810 23:00:59.031229 22567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } attempt: 1
I20260810 23:00:59.032044 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.031706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43674)
0810 23:00:59.031917 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.031978 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 23:00:59.032765 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.032550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43680)
0810 23:00:59.032660 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.032716 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 23:00:59.032938 22567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33419: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
I20260810 23:00:59.047134 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } attempt: 1
I20260810 23:00:59.047788 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.047557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43688)
0810 23:00:59.047682 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.047730 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 23:00:59.048436 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.048263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43694)
0810 23:00:59.048352 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.048394 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:00:59.048633 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33419: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
W20260810 23:00:59.102746 22655 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:00:59.102960 22655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:00:59.102980 22655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:00:59.103106 22655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:00:59.106720 22655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:00:59.106763 22655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:00:59.106781 22655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:00:59.106796 22655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:00:59.106840 22655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:00:59.108250 22567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } attempt: 2
I20260810 23:00:59.109004 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.108732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43706)
0810 23:00:59.108866 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.108926 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:00:59.109795 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.109535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43720)
0810 23:00:59.109676 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.109733 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 23:00:59.109992 22567 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33419: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
I20260810 23:00:59.111864 22655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:45131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:43725,127.29.78.189:33389,127.29.78.188:33419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:33419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:00:59.112668 22655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:00:59.113531 22655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:00:59.116796 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } attempt: 2
I20260810 23:00:59.117533 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.117214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43722)
0810 23:00:59.117366 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.117424 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 23:00:59.118176 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.117976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43734)
0810 23:00:59.118091 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.118132 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:00:59.118295 22646 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:33419: Network error: Client connection negotiation failed: client connection to 127.29.78.188:33419: connect: Connection refused (error 111)
W20260810 23:00:59.121307 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:59.121390 22655 server_base.cc:1034] running on GCE node
W20260810 23:00:59.121395 22661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:00:59.121305 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:00:59.122100 22655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:00:59.123066 22655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:00:59.124218 22655 hybrid_clock.cc:648] HybridClock initialized: now 1786402859124202 us; error 25 us; skew 500 ppm
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402859, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:00:59.126717 22655 init.cc:377] Logged in from keytab as kudu/127.29.78.188@KRBTEST.COM (short username kudu)
I20260810 23:00:59.127419 22655 webserver.cc:460] Webserver started at http://127.29.78.188:39135/ using document root <none> and password file <none>
I20260810 23:00:59.127846 22655 fs_manager.cc:359] Metadata directory not provided
I20260810 23:00:59.127905 22655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:00:59.128072 22655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:00:59.129900 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data/instance:
uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70"
format_stamp: "Formatted at 2026-08-10 23:00:59 on dist-test-slave-6nmv"
server_key: "9622b8998604b6206fcb75c7481baf73"
server_key_iv: "e0ff5f76a58ebe6ecc05fd5d1e88b160"
server_key_version: "encryptionkey@0"
I20260810 23:00:59.130393 22655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/wal/instance:
uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70"
format_stamp: "Formatted at 2026-08-10 23:00:59 on dist-test-slave-6nmv"
server_key: "9622b8998604b6206fcb75c7481baf73"
server_key_iv: "e0ff5f76a58ebe6ecc05fd5d1e88b160"
server_key_version: "encryptionkey@0"
I20260810 23:00:59.133785 22655 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 23:00:59.135984 22673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:00:59.137056 22655 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:00:59.137187 22655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/wal
uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70"
format_stamp: "Formatted at 2026-08-10 23:00:59 on dist-test-slave-6nmv"
server_key: "9622b8998604b6206fcb75c7481baf73"
server_key_iv: "e0ff5f76a58ebe6ecc05fd5d1e88b160"
server_key_version: "encryptionkey@0"
I20260810 23:00:59.137284 22655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:00:59.161350 22655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:00:59.164144 22655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:00:59.164359 22655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:00:59.170938 22655 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:33419
I20260810 23:00:59.171221 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:33419 every 8 connection(s)
I20260810 23:00:59.171860 22655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
I20260810 23:00:59.173640 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 22655
I20260810 23:00:59.173775 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402578782454-30010-0/minicluster-data/master-2/wal/instance
I20260810 23:00:59.174053 30010 external_mini_cluster.cc:1428] Setting key bc0892b3ac2e9c0a45e15fed62318559
I20260810 23:00:59.174563 22734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:00:59.178755 22734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260810 23:00:59.189963 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.175275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.29.78.188:33419)
0810 23:00:59.175629 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.175653 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.175685 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:00:59.176316 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.176331 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.176614 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.176867 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.177733 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.178712 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.179882 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.180616 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.180774 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.183775 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.183781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.183794 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.183822 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.186791 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.187335 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.187342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.187348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.187486 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.187798 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.187802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.187804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.188290 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.188492 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.188836 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.189094 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":133,"threads_started":1}
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402859, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:00:59.195350 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.180611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:43725 (local address 127.0.0.1:57944)
0810 23:00:59.181931 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.181991 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:59.182108 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:59.186594 (+ 4486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:59.186602 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:59.187043 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:59.187234 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.187244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.188442 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.188447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.189326 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.189336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.189833 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.189837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.189964 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.190573 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:59.190584 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:59.191998 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:59.193870 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.193884 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.193888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.194133 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.194411 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.194413 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.194416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.194505 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.194850 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:59.194855 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:59.194957 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 23:00:59.195098 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 23:00:59.195243 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":125,"threads_started":1}
I20260810 23:00:59.195436 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.184346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57944 (local address 127.29.78.190:43725)
0810 23:00:59.185101 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.185106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.185121 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:00:59.185184 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.185186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.185251 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.185342 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.187446 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.188281 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.189452 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.189690 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.189799 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.192137 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.192141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.192143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.192175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.193709 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.194236 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.194239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.194241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.194329 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.194593 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.194596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.194597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.194756 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.194825 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.195198 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.195299 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552}
I20260810 23:00:59.196341 22734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402859, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260810 23:00:59.202530 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.192281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57960 (local address 127.29.78.190:43725)
0810 23:00:59.192696 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.192706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.192719 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:00:59.192758 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.192762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.192807 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.192916 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.193609 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.194268 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.195143 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.195430 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.195538 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.198290 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.198294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.198297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.198320 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.200102 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.200638 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.200642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.200647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.200737 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.201563 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.201566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.201567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.201770 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.201967 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.202203 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.202324 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":131,"threads_started":1}
I20260810 23:00:59.205122 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.196806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:33389 (local address 127.0.0.1:49838)
0810 23:00:59.196911 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.196925 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:59.197005 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:59.197301 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:59.197304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:59.197477 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:59.197611 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.197618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.198494 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.198496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.198978 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.198986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.199866 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.199869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.199933 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.200346 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:59.200361 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:59.201591 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:59.203529 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.203532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.203534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.203781 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.204080 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.204085 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.204089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.204179 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.204708 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:59.204709 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:59.204770 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 23:00:59.204870 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 23:00:59.204996 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:00:59.205153 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.196904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.29.78.189:33389)
0810 23:00:59.197059 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.197064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.197081 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:00:59.197147 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.197150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.197189 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.197250 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.197762 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.198376 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.199365 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.199755 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.199875 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.201718 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.201721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.201722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.201745 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.203425 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.203872 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.203878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.203883 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.203998 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.204384 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.204389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.204393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.204621 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.204712 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.204954 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.205054 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:00:59.205739 22734 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260810 23:00:59.209143 22567 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } attempt: 3
I20260810 23:00:59.209502 22646 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } attempt: 3
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402859, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260810 23:00:59.213210 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.203650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49854 (local address 127.29.78.189:33389)
0810 23:00:59.203764 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.203769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.203781 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:00:59.203956 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.203958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.204016 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.204102 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.204852 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.205510 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.206503 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.206809 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.206897 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.209402 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.209409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.209414 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.209439 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.211208 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.211713 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.211716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.211718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.211799 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.212130 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.212134 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.212135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.212334 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.212524 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.212954 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.213054 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 10 23:00:59 dist-test-slave-6nmv krb5kdc[22478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402858, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260810 23:00:59.216854 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:00:59.217343 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.206158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43758)
0810 23:00:59.206245 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.206258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:59.206314 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:59.206518 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:59.206520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:59.206651 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:59.206719 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.206725 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.208492 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.208495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.209163 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.209173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.210248 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.210251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.210359 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.211088 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:59.211105 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:59.212439 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:59.215150 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.215152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.215153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.215382 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.216353 (+ 971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.216355 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.216357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.216445 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.216826 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:59.216830 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:59.216922 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 23:00:59.217009 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:00:59.217214 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":18}
I20260810 23:00:59.217859 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.206217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43758 (local address 127.29.78.188:33419)
0810 23:00:59.206315 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.206318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.206330 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:00:59.206380 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.206382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.206418 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.206484 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.206832 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.208383 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.209332 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.210137 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.210250 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.212587 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.212593 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.212598 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.212625 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.214715 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.215503 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.215508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.215512 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.215595 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.216548 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.216550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.216552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.216720 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.216797 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.217115 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.217652 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":395,"server-negotiator.queue_time_us":37}
I20260810 23:00:59.219822 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.209486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43772)
0810 23:00:59.209957 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.209973 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:59.210065 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:59.210679 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:59.210682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:59.210834 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:59.210927 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.210933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.211913 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.211916 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.212688 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.212697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.213305 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.213309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.213400 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.213798 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:59.213838 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:59.215252 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:59.217857 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.217862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.217865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.218106 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.218569 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.218572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.218574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.218663 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.219381 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:59.219384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:59.219480 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 23:00:59.219557 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:00:59.219672 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 23:00:59.218613 22539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57960
I20260810 23:00:59.220191 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.209937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:33419 (local address 127.0.0.1:43776)
0810 23:00:59.210482 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 23:00:59.210496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:00:59.210782 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:00:59.211202 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:00:59.211205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:00:59.211374 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:00:59.211458 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.211463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.212546 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.212549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.213499 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:00:59.213508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.214394 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.214398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:00:59.214487 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.215381 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:00:59.215394 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:00:59.216487 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:00:59.218569 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.218572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.218574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.218885 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.219206 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:00:59.219209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:00:59.219210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:00:59.219306 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:00:59.219724 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:00:59.219730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:00:59.219822 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 23:00:59.219903 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:00:59.220036 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260810 23:00:59.220413 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.209747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.29.78.188:33419)
0810 23:00:59.210373 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.210396 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.210408 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:00:59.210453 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.210457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.210516 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.210636 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.211037 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.211797 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.212819 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.213183 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.213277 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.215851 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.215854 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.215856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.215877 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.217710 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.218377 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.218381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.218384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.218456 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.219016 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.219020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.219021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.219247 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.220061 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.220139 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.220251 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":113,"threads_started":1}
I20260810 23:00:59.221140 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: Bootstrap starting.
I20260810 23:00:59.221271 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.209952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.29.78.188:33419)
0810 23:00:59.210955 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.210962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.210970 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:00:59.211028 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.211030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.211067 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.211550 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.211696 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.212423 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.213933 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.214276 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.214678 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.216628 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.216635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.216639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.216665 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.218464 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.219009 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.219012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.219013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.219099 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.219418 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.219421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.219423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.219607 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.220906 (+ 1299us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.220974 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.221080 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":84,"threads_started":1}
I20260810 23:00:59.223018 22567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91: Bootstrap starting.
I20260810 23:00:59.223733 22646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: Bootstrap starting.
I20260810 23:00:59.223819 22734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:59.224617 22734 log.cc:824] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:59.225123 22567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:59.225700 22646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: Neither blocks nor log segments found. Creating new log.
I20260810 23:00:59.225829 22567 log.cc:824] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:59.226579 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: No bootstrap required, opened a new log
I20260810 23:00:59.226574 22646 log.cc:824] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: Log is configured to *not* fsync() on all Append() calls
I20260810 23:00:59.227535 22567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91: No bootstrap required, opened a new log
I20260810 23:00:59.228027 22646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: No bootstrap required, opened a new log
I20260810 23:00:59.230398 22567 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.230674 22567 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a7dd9bf0cf4f0fa0c4e55f682e1c91, State: Initialized, Role: FOLLOWER
I20260810 23:00:59.230937 22646 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.230993 22567 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.231123 22646 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f821c400604a25a482d8993caee706, State: Initialized, Role: FOLLOWER
I20260810 23:00:59.231426 22646 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.231451 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.231395 22734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.231740 22734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 880a0f4cfb7c4e3ea96dd0eda6855a70, State: Initialized, Role: FOLLOWER
I20260810 23:00:59.231778 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:00:59.232309 22567 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:59.232338 22539 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:00:59.232426 22539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:59.232553 22646 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:59.233634 22539 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.235328 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.235478 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:00:59.234998 22619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d0f821c400604a25a482d8993caee706"
I20260810 23:00:59.236219 22619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:59.237424 22619 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a7dd9bf0cf4f0fa0c4e55f682e1c91 in term 1.
I20260810 23:00:59.237746 22539 leader_election.cc:288] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [CANDIDATE]: Term 1 election: Requested vote from peers d0f821c400604a25a482d8993caee706 (127.29.78.189:33389), 880a0f4cfb7c4e3ea96dd0eda6855a70 (127.29.78.188:33419)
I20260810 23:00:59.232191 22734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.238133 22509 leader_election.cc:302] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b0a7dd9bf0cf4f0fa0c4e55f682e1c91, d0f821c400604a25a482d8993caee706; no voters:
I20260810 23:00:59.238397 22747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [term 1 FOLLOWER]: Leader election won for term 1
W20260810 23:00:59.238826 22773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:00:59.238942 22773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:00:59.239292 22734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:00:59.239473 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.239697 22747 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [term 1 LEADER]: Becoming Leader. State: Replica: b0a7dd9bf0cf4f0fa0c4e55f682e1c91, State: Running, Role: LEADER
I20260810 23:00:59.240008 22747 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } }
I20260810 23:00:59.239567 22705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70"
I20260810 23:00:59.241568 22705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:00:59.242105 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a7dd9bf0cf4f0fa0c4e55f682e1c91. Latest consensus state: current_term: 1 leader_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.242235 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: This master's current role is: LEADER
W20260810 23:00:59.237771 22771 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:00:59.242625 22771 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:00:59.242813 22705 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0a7dd9bf0cf4f0fa0c4e55f682e1c91 in term 1.
I20260810 23:00:59.243353 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:00:59.244135 22514 tablet.cc:2369] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:00:59.241633 22574 master_runner.cc:186] Error getting master registration for 127.29.78.188:33419: OK, instance_id { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" instance_seqno: 1786402859169093 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:00:59.244890 22574 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:00:59.246765 22798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:00:59.247648 22798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:00:59.249557 22492 master_runner.cc:263] No leader master found from master b0a7dd9bf0cf4f0fa0c4e55f682e1c91
W20260810 23:00:59.249586 22799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:00:59.249626 22492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:00:59.249661 22799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:00:59.257352 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.239936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.29.78.189:33389)
0810 23:00:59.240062 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.240068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.246058 (+ 5990us) server_negotiation.cc:407] Connection header received
0810 23:00:59.246515 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.246518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.246560 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.246605 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.249829 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.250598 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.251737 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.252074 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.252151 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.253986 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.253990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.253992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.254016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.255619 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.256088 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.256092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.256094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.256173 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.256455 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.256458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.256471 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.256675 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.256793 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.257119 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.257219 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 23:00:59.265264 22619 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [term 1 FOLLOWER]: Refusing update from remote peer b0a7dd9bf0cf4f0fa0c4e55f682e1c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:59.265929 22705 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [term 1 FOLLOWER]: Refusing update from remote peer b0a7dd9bf0cf4f0fa0c4e55f682e1c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:00:59.266706 22782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:59.267105 22782 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:00:59.270778 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a7dd9bf0cf4f0fa0c4e55f682e1c91. Latest consensus state: current_term: 1 leader_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.270889 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:00:59.272919 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a7dd9bf0cf4f0fa0c4e55f682e1c91. Latest consensus state: current_term: 1 leader_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.273032 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:00:59.280999 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.240894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.29.78.190:43725)
0810 23:00:59.241361 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.241418 (+ 57us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.246264 (+ 4846us) server_negotiation.cc:407] Connection header received
0810 23:00:59.246390 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.246392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.246430 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.246514 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.257899 (+ 11385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.258580 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.260722 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.261712 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.261790 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.274280 (+ 12490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.274284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.274286 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.274310 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.275789 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.278522 (+ 2733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.278525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.278527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.278615 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.279319 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.279322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.279324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.279570 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.279712 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.280760 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.280872 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260810 23:00:59.281673 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.281798 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:59.282006 22801 mvcc.cc:205] Tried to move back new op lower bound from 7317106111544455168 to 7317106111450710016. Current Snapshot: MvccSnapshot[applied={T|T < 7317106111544455168}]
I20260810 23:00:59.282153 22748 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.282260 22748 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:00:59.282202 22807 mvcc.cc:205] Tried to move back new op lower bound from 7317106111544455168 to 7317106111450710016. Current Snapshot: MvccSnapshot[applied={T|T < 7317106111544455168}]
I20260810 23:00:59.283928 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:00:59.240664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.29.78.188:33419)
0810 23:00:59.242585 (+ 1921us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.242591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.247937 (+ 5346us) server_negotiation.cc:407] Connection header received
0810 23:00:59.248071 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.248074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.248132 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.248207 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.252001 (+ 3794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.252733 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.262539 (+ 9806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:00:59.262888 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:00:59.262985 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:00:59.276229 (+ 13244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:00:59.276233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:00:59.276235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:00:59.276257 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:00:59.277682 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.278938 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.278941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.278942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.279023 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:00:59.280033 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:00:59.280039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:00:59.280044 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:00:59.280252 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:00:59.280383 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:00:59.283674 (+ 3291us) server_negotiation.cc:299] Negotiation successful
0810 23:00:59.283793 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1850}
I20260810 23:00:59.287696 22782 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.287806 22782 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91 [sys.catalog]: This master's current role is: LEADER
I20260810 23:00:59.288781 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a7dd9bf0cf4f0fa0c4e55f682e1c91" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 43725 } } peers { permanent_uuid: "d0f821c400604a25a482d8993caee706" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 33389 } } peers { permanent_uuid: "880a0f4cfb7c4e3ea96dd0eda6855a70" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 33419 } } }
I20260810 23:00:59.288888 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:00:59.289005 22805 mvcc.cc:205] Tried to move back new op lower bound from 7317106111544455168 to 7317106111450710016. Current Snapshot: MvccSnapshot[applied={T|T < 7317106111544455168}]
I20260810 23:00:59.290910 22798 catalog_manager.cc:1347] Generated new cluster ID: 7419b4611d594bb1b95f5b1b9ba4a7d4
I20260810 23:00:59.290975 22798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:00:59.333106 22798 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:00:59.334331 22798 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:00:59.347402 22798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0a7dd9bf0cf4f0fa0c4e55f682e1c91: Generated new TSK 0
I20260810 23:00:59.348061 22798 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:00:59.380263 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.377415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.29.78.189:33389)
0810 23:00:59.377529 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.377534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.379218 (+ 1684us) server_negotiation.cc:407] Connection header received
0810 23:00:59.379422 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.379426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.379472 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.379524 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.380136 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49876: BlockingRecv error: recv got EOF from 127.0.0.1:49876 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 23:00:59.381484 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.377891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.29.78.190:43725)
0810 23:00:59.377990 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.377994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.380689 (+ 2695us) server_negotiation.cc:407] Connection header received
0810 23:00:59.380827 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.380829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.380869 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.380923 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.381361 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57968: BlockingRecv error: recv got EOF from 127.0.0.1:57968 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 23:00:59.382778 22742 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.378485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.29.78.188:33419)
0810 23:00:59.378585 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.378589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.381991 (+ 3402us) server_negotiation.cc:407] Connection header received
0810 23:00:59.382143 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.382146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.382189 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.382235 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.382674 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43792: BlockingRecv error: recv got EOF from 127.0.0.1:43792 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 23:00:59.387969 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.386542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.29.78.189:33389)
0810 23:00:59.386640 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.386644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.387066 (+ 422us) server_negotiation.cc:407] Connection header received
0810 23:00:59.387253 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.387256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.387318 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.387365 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.387882 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49878: BlockingRecv error: recv got EOF from 127.0.0.1:49878 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260810 23:00:59.392581 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.391195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.29.78.189:33389)
0810 23:00:59.391275 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.391278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.391806 (+ 528us) server_negotiation.cc:407] Connection header received
0810 23:00:59.391977 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.391980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.392022 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.392066 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.392506 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49880: BlockingRecv error: recv got EOF from 127.0.0.1:49880 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 23:00:59.397698 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.396371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.29.78.189:33389)
0810 23:00:59.396452 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.396455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.396958 (+ 503us) server_negotiation.cc:407] Connection header received
0810 23:00:59.397119 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.397121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.397161 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.397205 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.397624 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49894: BlockingRecv error: recv got EOF from 127.0.0.1:49894 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 23:00:59.405079 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.403737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.29.78.189:33389)
0810 23:00:59.403897 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.403902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.404336 (+ 434us) server_negotiation.cc:407] Connection header received
0810 23:00:59.404495 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.404497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.404543 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.404583 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.404999 (+ 416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49896: BlockingRecv error: recv got EOF from 127.0.0.1:49896 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 23:00:59.416441 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.415094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.29.78.189:33389)
0810 23:00:59.415181 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.415185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.415602 (+ 417us) server_negotiation.cc:407] Connection header received
0810 23:00:59.415771 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.415773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.415815 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.415860 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.416354 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49904: BlockingRecv error: recv got EOF from 127.0.0.1:49904 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 23:00:59.436067 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.434588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.29.78.189:33389)
0810 23:00:59.434685 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.434690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.435224 (+ 534us) server_negotiation.cc:407] Connection header received
0810 23:00:59.435414 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.435418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.435466 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.435510 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.435973 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49920: BlockingRecv error: recv got EOF from 127.0.0.1:49920 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 23:00:59.471851 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.470519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49928 (local address 127.29.78.189:33389)
0810 23:00:59.470665 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.470671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.471000 (+ 329us) server_negotiation.cc:407] Connection header received
0810 23:00:59.471063 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.471065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.471115 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.471194 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.471719 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49928: BlockingRecv error: recv got EOF from 127.0.0.1:49928 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 23:00:59.539031 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.537835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.29.78.189:33389)
0810 23:00:59.537997 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.538002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.538154 (+ 152us) server_negotiation.cc:407] Connection header received
0810 23:00:59.538319 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.538322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.538377 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.538469 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.538911 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49942: BlockingRecv error: recv got EOF from 127.0.0.1:49942 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 23:00:59.670286 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.669149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.29.78.189:33389)
0810 23:00:59.669334 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.669338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.669403 (+ 65us) server_negotiation.cc:407] Connection header received
0810 23:00:59.669535 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.669537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.669587 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.669660 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.670129 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49958: BlockingRecv error: recv got EOF from 127.0.0.1:49958 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:00:59.930008 22652 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:00:59.928662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.29.78.189:33389)
0810 23:00:59.928794 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:00:59.928798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:00:59.928973 (+ 175us) server_negotiation.cc:407] Connection header received
0810 23:00:59.929134 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:00:59.929137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:00:59.929192 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:00:59.929277 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:00:59.929876 (+ 599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49964: BlockingRecv error: recv got EOF from 127.0.0.1:49964 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:01:00.240279 22773 catalog_manager.cc:1259] Loaded cluster ID: 7419b4611d594bb1b95f5b1b9ba4a7d4
I20260810 23:01:00.240360 22773 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: loading cluster ID for follower catalog manager: success
I20260810 23:01:00.242085 22773 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: acquiring CA information for follower catalog manager: success
I20260810 23:01:00.243136 22773 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0f821c400604a25a482d8993caee706: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:01:00.250757 22799 catalog_manager.cc:1259] Loaded cluster ID: 7419b4611d594bb1b95f5b1b9ba4a7d4
I20260810 23:01:00.250842 22799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: loading cluster ID for follower catalog manager: success
I20260810 23:01:00.252347 22799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: acquiring CA information for follower catalog manager: success
I20260810 23:01:00.253338 22799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 880a0f4cfb7c4e3ea96dd0eda6855a70: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:01:00.447777 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:00.444076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.29.78.189:33389)
0810 23:01:00.444370 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:00.444373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:00.444390 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:01:00.444483 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:00.444486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:00.444614 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:00.444710 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:00.445006 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.445515 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.446431 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.446671 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.446753 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:00.447028 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:00.447445 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:00.447547 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:00.447599 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:01:00.447650 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":94,"threads_started":1}
I20260810 23:01:00.454602 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:00.450951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.29.78.188:33419)
0810 23:01:00.451216 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:00.451220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:00.451306 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:01:00.451430 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:00.451432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:00.451547 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:00.451668 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:00.451930 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.452440 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.453240 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.453452 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.453534 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:00.453772 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:00.454198 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:00.454286 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:00.454352 (+ 66us) server_negotiation.cc:299] Negotiation successful
0810 23:01:00.454398 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
I20260810 23:01:00.461699 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:00.457636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.29.78.189:33389)
0810 23:01:00.457758 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:00.457762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:00.458349 (+ 587us) server_negotiation.cc:407] Connection header received
0810 23:01:00.458402 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:00.458404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:00.458451 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:00.458538 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:00.458899 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.459681 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.460618 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.460837 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.460948 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:00.461193 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:00.461277 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:00.461346 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:00.461504 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:01:00.461556 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:01:00.462455 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:00.458060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.29.78.188:33419)
0810 23:01:00.458172 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:00.458175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:00.458888 (+ 713us) server_negotiation.cc:407] Connection header received
0810 23:01:00.459020 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:00.459022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:00.459059 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:00.459139 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:00.459376 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.459931 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.461078 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.461362 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.461474 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:00.461789 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:00.461887 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:00.461967 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:00.462279 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 23:01:00.462328 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:01:00.462898 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:00.457611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.29.78.190:43725)
0810 23:01:00.458204 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:00.458208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:00.458219 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:00.458293 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:00.458295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:00.458469 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:00.458584 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:00.459072 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.459822 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.461118 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:00.461463 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:00.461592 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:00.462089 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:00.462372 (+ 283us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:00.462455 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:00.462737 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 23:01:00.462796 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":83,"threads_started":1}
I20260810 23:01:00.465951 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 22492
I20260810 23:01:00.476843 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 22574
I20260810 23:01:00.484115 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 22655
2026-08-10T23:01:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3075 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:01:00 dist-test-slave-6nmv krb5kdc[22908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:01:00 dist-test-slave-6nmv krb5kdc[22908](info): set up 2 sockets
Aug 10 23:01:00 dist-test-slave-6nmv krb5kdc[22908](info): commencing operation
krb5kdc: starting...
W20260810 23:01:01.524811 30010 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/test-user.keytab.
Aug 10 23:01:01 dist-test-slave-6nmv krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:01:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.29.78.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
Entry for principal kudu/127.29.78.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab.
I20260810 23:01:01.649310 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39429
--webserver_interface=127.29.78.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39429,127.29.78.189:39255,127.29.78.188:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:01:01.759601 22923 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:01:01.759915 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:01.759975 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:01.760053 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:01:01.764015 22923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:01.764108 22923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:01.764147 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:01.764184 22923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:01.764221 22923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:01.769531 22923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39429,127.29.78.189:39255,127.29.78.188:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.190.keytab
--principal=kudu/127.29.78.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.190:39429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.29.78.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:01.770582 22923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:01.771559 22923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:01.779951 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:01.779951 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:01.779968 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:01.780314 22923 server_base.cc:1034] running on GCE node
I20260810 23:01:01.780894 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:01.782019 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:01.783200 22923 hybrid_clock.cc:648] HybridClock initialized: now 1786402861783183 us; error 28 us; skew 500 ppm
Aug 10 23:01:01 dist-test-slave-6nmv krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:01.785980 22923 init.cc:377] Logged in from keytab as kudu/127.29.78.190@KRBTEST.COM (short username kudu)
I20260810 23:01:01.786851 22923 webserver.cc:460] Webserver started at http://127.29.78.190:34875/ using document root <none> and password file <none>
I20260810 23:01:01.787380 22923 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:01.787480 22923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:01.787703 22923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:01.789614 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data/instance:
uuid: "1d3d0d4795104adc8981eb333346e316"
format_stamp: "Formatted at 2026-08-10 23:01:01 on dist-test-slave-6nmv"
server_key: "e12e970e1226d05f49e787310f090636"
server_key_iv: "815d0bb2419bb82007024867a631b274"
server_key_version: "encryptionkey@0"
I20260810 23:01:01.790226 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/wal/instance:
uuid: "1d3d0d4795104adc8981eb333346e316"
format_stamp: "Formatted at 2026-08-10 23:01:01 on dist-test-slave-6nmv"
server_key: "e12e970e1226d05f49e787310f090636"
server_key_iv: "815d0bb2419bb82007024867a631b274"
server_key_version: "encryptionkey@0"
I20260810 23:01:01.793731 22923 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 23:01:01.795873 22947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:01.797111 22923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:01:01.797273 22923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/wal
uuid: "1d3d0d4795104adc8981eb333346e316"
format_stamp: "Formatted at 2026-08-10 23:01:01 on dist-test-slave-6nmv"
server_key: "e12e970e1226d05f49e787310f090636"
server_key_iv: "815d0bb2419bb82007024867a631b274"
server_key_version: "encryptionkey@0"
I20260810 23:01:01.797385 22923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:01.818070 22923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:01.820658 22923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:01.820858 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:01.827759 22923 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.190:39429
I20260810 23:01:01.828018 23010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.190:39429 every 8 connection(s)
I20260810 23:01:01.828656 22923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/data/info.pb
I20260810 23:01:01.831269 23012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:01.834868 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:01.835585 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 22923
I20260810 23:01:01.835726 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-0/wal/instance
I20260810 23:01:01.835984 30010 external_mini_cluster.cc:1428] Setting key cb04bd24380cfa7563cdad1b25232c1c
WARNING: no policy specified for kudu/127.29.78.189@KRBTEST.COM; defaulting to no policy
Aug 10 23:01:01 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.189@KRBTEST.COM" created.
I20260810 23:01:01.849139 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.837075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57424 (local address 127.29.78.190:39429)
0810 23:01:01.837463 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:01.837492 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:01.837753 (+ 261us) server_negotiation.cc:407] Connection header received
0810 23:01:01.838763 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:01.838767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:01.838938 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:01.839109 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:01.839940 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:01.840798 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:01.841729 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:01.842144 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:01.842429 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:01.844689 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:01.844695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:01.844706 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:01.844734 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:01.846234 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:01.846730 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:01.846741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:01.846743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:01.846838 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:01.847151 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:01.847156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:01.847158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:01.847486 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:01.847617 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:01.847961 (+ 344us) server_negotiation.cc:299] Negotiation successful
0810 23:01:01.848209 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":256,"thread_start_us":127,"threads_started":1}
I20260810 23:01:01.849139 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.837076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39429 (local address 127.0.0.1:57424)
0810 23:01:01.837616 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:01.837649 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:01.838479 (+ 830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:01.839109 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:01.839113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:01.839559 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:01.839780 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:01.839793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:01.840919 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:01.840923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:01.841601 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:01.841610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:01.842289 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:01.842293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:01.842524 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:01.843089 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:01.843104 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:01.844557 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:01.846337 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:01.846347 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:01.846355 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:01.846620 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:01.846937 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:01.846940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:01.846942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:01.847062 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:01.847602 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:01.847607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:01.847739 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 23:01:01.847890 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 23:01:01.848257 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":186,"spinlock_wait_cycles":8448,"thread_start_us":143,"threads_started":1}
I20260810 23:01:01.851382 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:01.852228 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.851895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44664)
0810 23:01:01.852049 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:01.852144 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 23:01:01.852444 22949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
I20260810 23:01:01.853308 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.853061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44666)
0810 23:01:01.853184 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:01.853240 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 23:01:01.853469 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39255: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
Entry for principal kudu/127.29.78.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab.
I20260810 23:01:01.862536 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:39255
--webserver_interface=127.29.78.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39429,127.29.78.189:39255,127.29.78.188:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:01:01.902719 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } attempt: 1
I20260810 23:01:01.903666 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.903307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44668)
0810 23:01:01.903492 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:01.903569 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260810 23:01:01.904500 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.904196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44670)
0810 23:01:01.904333 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:01.904411 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:01:01.904655 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39255: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
W20260810 23:01:01.971943 23020 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:01:01.972182 23020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:01.972204 23020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:01.972239 23020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:01:01.972832 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } attempt: 2
I20260810 23:01:01.973613 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.973305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44686)
0810 23:01:01.973465 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:01.973529 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:01:01.974291 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:01.974093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44692)
0810 23:01:01.974192 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:01.974237 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 23:01:01.974409 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.189:39255: Network error: Client connection negotiation failed: client connection to 127.29.78.189:39255: connect: Connection refused (error 111)
W20260810 23:01:01.976344 23020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:01.976387 23020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:01.976403 23020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:01.976418 23020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:01.976431 23020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:01.981375 23020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39429,127.29.78.189:39255,127.29.78.188:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.189.keytab
--principal=kudu/127.29.78.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.189:39255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.29.78.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:01.982318 23020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:01.983207 23020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:01:01.990811 23032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:01.991004 23020 server_base.cc:1034] running on GCE node
W20260810 23:01:01.990814 23030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:01.990803 23029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:01.991693 23020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:01.992697 23020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:01.993896 23020 hybrid_clock.cc:648] HybridClock initialized: now 1786402861993866 us; error 40 us; skew 500 ppm
Aug 10 23:01:01 dist-test-slave-6nmv krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:01.996564 23020 init.cc:377] Logged in from keytab as kudu/127.29.78.189@KRBTEST.COM (short username kudu)
I20260810 23:01:01.997402 23020 webserver.cc:460] Webserver started at http://127.29.78.189:44917/ using document root <none> and password file <none>
I20260810 23:01:01.997998 23020 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:01.998061 23020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:01.998296 23020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:02.000255 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data/instance:
uuid: "8e4c08c67d724bc2a7920eda29998f2b"
format_stamp: "Formatted at 2026-08-10 23:01:01 on dist-test-slave-6nmv"
server_key: "e061b172f66870a46bf4fcf97e5054c5"
server_key_iv: "196eed1cf9f472048b9abdc06bf04fb9"
server_key_version: "encryptionkey@0"
I20260810 23:01:02.000820 23020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/wal/instance:
uuid: "8e4c08c67d724bc2a7920eda29998f2b"
format_stamp: "Formatted at 2026-08-10 23:01:01 on dist-test-slave-6nmv"
server_key: "e061b172f66870a46bf4fcf97e5054c5"
server_key_iv: "196eed1cf9f472048b9abdc06bf04fb9"
server_key_version: "encryptionkey@0"
I20260810 23:01:02.004433 23020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 23:01:02.006757 23041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:02.008096 23020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 23:01:02.008193 23020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/wal
uuid: "8e4c08c67d724bc2a7920eda29998f2b"
format_stamp: "Formatted at 2026-08-10 23:01:01 on dist-test-slave-6nmv"
server_key: "e061b172f66870a46bf4fcf97e5054c5"
server_key_iv: "196eed1cf9f472048b9abdc06bf04fb9"
server_key_version: "encryptionkey@0"
I20260810 23:01:02.008312 23020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:02.031685 23020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:02.034787 23020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:02.034948 23020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:02.041604 23020 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.189:39255
I20260810 23:01:02.041971 23099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.189:39255 every 8 connection(s)
I20260810 23:01:02.042662 23020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/data/info.pb
I20260810 23:01:02.045933 23100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:02.049045 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 23020
I20260810 23:01:02.049206 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-1/wal/instance
I20260810 23:01:02.049515 30010 external_mini_cluster.cc:1428] Setting key ca4b9b58dc425a8e41ded6d3547a7eef
I20260810 23:01:02.050071 23100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.29.78.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.29.78.188@KRBTEST.COM" created.
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:01:02.065569 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } attempt: 3
I20260810 23:01:02.068768 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.052047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.29.78.190:39429)
0810 23:01:02.052210 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.052215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.053082 (+ 867us) server_negotiation.cc:407] Connection header received
0810 23:01:02.054077 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.054081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.054146 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.054254 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.055774 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.056293 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.057938 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.058156 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.058229 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.063016 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.063018 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.063020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.063045 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.065164 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.065860 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.065865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.065867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.065955 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.066405 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.066410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.066411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.066677 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.066781 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.068474 (+ 1693us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.068604 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:01:02.069046 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.052415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39429 (local address 127.0.0.1:57436)
0810 23:01:02.052931 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.052981 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.053807 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.054420 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.054427 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.054883 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.055584 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.055608 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.056423 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.056428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.057759 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.057771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.058307 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.058311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.058507 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.059117 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.059137 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.062842 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.065289 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.065294 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.065312 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.065654 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.066104 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.066109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.066111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.066275 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.066809 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.066820 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.067217 (+ 397us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.067471 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.067883 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260810 23:01:02.071035 23100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260810 23:01:02.078392 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.066100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44700)
0810 23:01:02.066335 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.066353 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.066440 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.066997 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.067001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.067192 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.067292 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.067297 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.068360 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.068366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.069192 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.069202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.070020 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.070024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.070143 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.070597 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.070611 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.074028 (+ 3417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.076350 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.076353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.076356 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.076622 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.077054 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.077057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.077058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.077144 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.077739 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.077741 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.077886 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.078022 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.078181 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260810 23:01:02.078490 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.066228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44700 (local address 127.29.78.189:39255)
0810 23:01:02.066578 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.066591 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.066614 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:01:02.066654 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.066659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.066872 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.067026 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.067419 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.068236 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.069324 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.069888 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.070360 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.074191 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.074199 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.074208 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.074226 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.076244 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.076786 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.076790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.076791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.076929 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.077246 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.077251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.077253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.077631 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.077773 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.078143 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.078292 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":152,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
Entry for principal kudu/127.29.78.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab.
I20260810 23:01:02.080309 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.071523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44710)
0810 23:01:02.071610 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.071621 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.071697 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.072145 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.072148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.072302 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.072376 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.072381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.073252 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.073256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.074018 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.074028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.074467 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.074470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.074571 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.074985 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.074996 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.076254 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.078869 (+ 2615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.078875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.078879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.079067 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.079336 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.079339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.079340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.079417 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.079787 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.079790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.079867 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.079987 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.080121 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"spinlock_wait_cycles":10496}
I20260810 23:01:02.080350 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.071661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44710 (local address 127.29.78.189:39255)
0810 23:01:02.071936 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.071942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.071953 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:02.072005 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.072007 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.072039 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.072136 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.072533 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.073135 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.074140 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.074364 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.074469 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.076406 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.076411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.076413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.076436 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.078768 (+ 2332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.079159 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.079165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.079167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.079243 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.079506 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.079528 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.079530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.079691 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.079798 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.080034 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.080175 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":226,"thread_start_us":104,"threads_started":1}
I20260810 23:01:02.080567 23012 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:02.081328 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.081099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34474)
0810 23:01:02.081224 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.081272 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:01:02.081429 23100 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:02.081720 30010 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskX35aNT/build/debug/bin/kudu
/tmp/dist-test-taskX35aNT/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:35769
--webserver_interface=127.29.78.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.29.78.148:39015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.78.190:39429,127.29.78.189:39255,127.29.78.188:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:01:02.082168 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.081868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34490)
0810 23:01:02.082009 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.082094 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 23:01:02.082366 23042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
I20260810 23:01:02.083168 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.082896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34502)
0810 23:01:02.083035 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.083095 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 23:01:02.083210 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.082178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34496)
0810 23:01:02.083120 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.083162 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":881}
W20260810 23:01:02.083386 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:35769: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
W20260810 23:01:02.083474 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:35769: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
I20260810 23:01:02.114552 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } attempt: 1
I20260810 23:01:02.115268 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.115019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34514)
0810 23:01:02.115155 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.115206 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:01:02.115958 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.115786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34518)
0810 23:01:02.115873 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.115911 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 23:01:02.116117 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:35769: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
I20260810 23:01:02.132773 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } attempt: 1
I20260810 23:01:02.133509 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.133220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34520)
0810 23:01:02.133372 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.133429 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:01:02.134289 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.134102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34534)
0810 23:01:02.134200 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.134243 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 23:01:02.134482 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:35769: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
W20260810 23:01:02.190157 23109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:01:02.190393 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:01:02.190415 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:01:02.190450 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 23:01:02.191277 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } attempt: 2
I20260810 23:01:02.192133 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.191851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34546)
0810 23:01:02.192001 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.192059 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 23:01:02.192929 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.192655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34558)
0810 23:01:02.192807 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.192876 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 23:01:02.193145 23012 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:35769: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
W20260810 23:01:02.194908 23109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:01:02.194977 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:01:02.194996 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:01:02.195011 23109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:01:02.195065 23109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:01:02.200124 23109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.78.148:39015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.29.78.190:39429,127.29.78.189:39255,127.29.78.188:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/krb5kdc/kudu_127.29.78.188.keytab
--principal=kudu/127.29.78.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.78.188:35769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.29.78.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 22:18:12 UTC on ccycloud.quasar-kzngpw.root.comops.site
build id 4579240
I20260810 23:01:02.201146 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:01:02.202224 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:01:02.202648 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } attempt: 2
I20260810 23:01:02.203339 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.203088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34568)
0810 23:01:02.203221 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.203273 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 23:01:02.204038 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.203818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34584)
0810 23:01:02.203939 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.203991 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 23:01:02.204190 23100 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.29.78.188:35769: Network error: Client connection negotiation failed: client connection to 127.29.78.188:35769: connect: Connection refused (error 111)
W20260810 23:01:02.210620 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:02.210613 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:01:02.210613 23122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:01:02.210857 23109 server_base.cc:1034] running on GCE node
I20260810 23:01:02.211373 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:01:02.212482 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:01:02.213673 23109 hybrid_clock.cc:648] HybridClock initialized: now 1786402862213651 us; error 33 us; skew 500 ppm
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:01:02.216436 23109 init.cc:377] Logged in from keytab as kudu/127.29.78.188@KRBTEST.COM (short username kudu)
I20260810 23:01:02.217276 23109 webserver.cc:460] Webserver started at http://127.29.78.188:41197/ using document root <none> and password file <none>
I20260810 23:01:02.217865 23109 fs_manager.cc:359] Metadata directory not provided
I20260810 23:01:02.217983 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:01:02.218211 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:01:02.220127 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data/instance:
uuid: "1b6089838641421ba3bada08ba88096a"
format_stamp: "Formatted at 2026-08-10 23:01:02 on dist-test-slave-6nmv"
server_key: "0c8a16f96d4cf0e68dfd517646970338"
server_key_iv: "5c3d1409952b96f9bdfaa29e4719744a"
server_key_version: "encryptionkey@0"
I20260810 23:01:02.220679 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/wal/instance:
uuid: "1b6089838641421ba3bada08ba88096a"
format_stamp: "Formatted at 2026-08-10 23:01:02 on dist-test-slave-6nmv"
server_key: "0c8a16f96d4cf0e68dfd517646970338"
server_key_iv: "5c3d1409952b96f9bdfaa29e4719744a"
server_key_version: "encryptionkey@0"
I20260810 23:01:02.224123 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 23:01:02.226420 23129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:01:02.227567 23109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:01:02.227712 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data,/tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/wal
uuid: "1b6089838641421ba3bada08ba88096a"
format_stamp: "Formatted at 2026-08-10 23:01:02 on dist-test-slave-6nmv"
server_key: "0c8a16f96d4cf0e68dfd517646970338"
server_key_iv: "5c3d1409952b96f9bdfaa29e4719744a"
server_key_version: "encryptionkey@0"
I20260810 23:01:02.227810 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:01:02.252594 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:01:02.255934 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:01:02.256155 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:01:02.263360 23109 rpc_server.cc:307] RPC server started. Bound to: 127.29.78.188:35769
I20260810 23:01:02.263644 23189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.78.188:35769 every 8 connection(s)
I20260810 23:01:02.264364 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/data/info.pb
I20260810 23:01:02.266963 23190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:01:02.268545 30010 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskX35aNT/build/debug/bin/kudu as pid 23109
I20260810 23:01:02.268695 30010 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskX35aNT/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402578782454-30010-0/minicluster-data/master-2/wal/instance
I20260810 23:01:02.269019 30010 external_mini_cluster.cc:1428] Setting key 26a03cd34766dacca7d77b5c6cbd2912
I20260810 23:01:02.271299 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:01:02.282393 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.273199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.29.78.190:39429)
0810 23:01:02.273329 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.273335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.273611 (+ 276us) server_negotiation.cc:407] Connection header received
0810 23:01:02.273773 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.273781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.273875 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.273961 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.274689 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.275201 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.276224 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.276547 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.276671 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.279024 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.279033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.279036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.279060 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.280424 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.280913 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.280916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.280918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.281002 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.281313 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.281317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.281319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.281506 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.281637 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.282087 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.282216 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:02.283295 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.270288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.29.78.188:35769)
0810 23:01:02.270683 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.270699 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.270734 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:01:02.271345 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.271360 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.271749 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.272024 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.272906 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.274261 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.275203 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.275707 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.275823 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.278587 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.278592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.278602 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.278624 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.280412 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.280862 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.280868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.280870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.280975 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.281259 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.281262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.281264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.281632 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.281885 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.282075 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.282360 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":258,"thread_start_us":135,"threads_started":1}
I20260810 23:01:02.283299 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.273142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.190:39429 (local address 127.0.0.1:57450)
0810 23:01:02.273489 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.273526 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.273660 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.274019 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.274025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.274394 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.274546 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.274554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.275323 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.275326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.276059 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.276070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.276704 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.276707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.276790 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.277324 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.277353 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.278866 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.280551 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.280562 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.280570 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.280809 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.281095 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.281100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.281102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.281218 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.281635 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.281643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.281780 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.282016 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.282538 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":181,"thread_start_us":107,"threads_started":1}
I20260810 23:01:02.284562 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.190@KRBTEST.COM
I20260810 23:01:02.292294 23012 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } attempt: 3
I20260810 23:01:02.293349 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.285181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44718 (local address 127.29.78.189:39255)
0810 23:01:02.285345 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.285349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.285362 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:01:02.285405 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.285408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.285448 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.285549 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.286000 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.286583 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.287371 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.287595 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.287673 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.289835 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.289842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.289847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.289874 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.291311 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.291828 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.291832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.291834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.291931 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.292281 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.292288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.292290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.292533 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.292644 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.292981 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.293147 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 23:01:02.293857 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.285086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.189:39255 (local address 127.0.0.1:44718)
0810 23:01:02.285239 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.285253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.285332 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.285565 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.285568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.285748 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.285868 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.285874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.286698 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.286702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.287215 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.287223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.287702 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.287705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.287789 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.288314 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.288324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.289675 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.291416 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.291419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.291420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.291724 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.292041 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.292043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.292044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.292159 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.292651 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.292654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.292758 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.293505 (+ 747us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.293663 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 23:01:02.294564 23190 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:01:02.295321 23100 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } attempt: 3
I20260810 23:01:02.295562 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.285720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.29.78.190:39429)
0810 23:01:02.285902 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.285905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.285916 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:02.285960 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.285963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.286005 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.286087 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.286757 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.287193 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.288135 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.288418 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.288499 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.291234 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.291240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.291243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.291272 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.293245 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.294160 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.294163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.294165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.294253 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.294549 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.294552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.294554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.294784 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.295045 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.295137 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.295249 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.190@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.189@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402862, etypes {rep=17 tkt=17 ses=17}, kudu/127.29.78.188@KRBTEST.COM for kudu/127.29.78.188@KRBTEST.COM
I20260810 23:01:02.302711 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.292715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34598)
0810 23:01:02.292862 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.292880 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.292956 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.293352 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.293356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.293534 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.293644 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.293649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.294746 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.294750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.295519 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.295531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.296432 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.296435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.296544 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.297417 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.297432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.298945 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.300837 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.300840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.300842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.301110 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.301502 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.301506 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.301508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.301602 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.302162 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.302168 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.302292 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.302394 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.302527 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 23:01:02.303375 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.292980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.29.78.188:35769)
0810 23:01:02.293132 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.293136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.293148 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:01:02.293195 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.293198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.293237 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.293314 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.293752 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.294618 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.295950 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.296318 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.296417 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.299089 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.299092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.299094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.299117 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.300729 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.301221 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.301224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.301226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.301328 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.301713 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.301718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.301723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.302014 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.302115 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.303050 (+ 935us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.303190 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57,"spinlock_wait_cycles":4736}
Aug 10 23:01:02 dist-test-slave-6nmv krb5kdc[22908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.29.78.189@KRBTEST.COM
I20260810 23:01:02.306445 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.295679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34618)
0810 23:01:02.295826 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.295837 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.295923 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.296939 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.296944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.297131 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.297249 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.297257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.298159 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.298165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.299034 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.299048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.299788 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.299792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.299917 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.300334 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.300349 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.302130 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.304792 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.304798 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.304802 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.305070 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.305380 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.305383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.305385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.305467 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.305963 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.305966 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.306072 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.306180 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.306301 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":24}
I20260810 23:01:02.306968 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.295701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.29.78.188:35769)
0810 23:01:02.296682 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.296692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.296704 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:01:02.296743 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.296747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.296788 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.296887 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.297370 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.298037 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.299176 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.299662 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.301190 (+ 1528us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.302470 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.302476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.302493 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.302574 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.304667 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.305198 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.305201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.305203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.305279 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.305596 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.305599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.305601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.305844 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.306619 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.306705 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.306810 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1092,"server-negotiator.queue_time_us":910,"spinlock_wait_cycles":17152,"thread_start_us":150,"threads_started":1}
I20260810 23:01:02.307227 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316: Bootstrap starting.
I20260810 23:01:02.307886 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.295026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.78.188:35769 (local address 127.0.0.1:34614)
0810 23:01:02.295424 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 23:01:02.295435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:01:02.295503 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:01:02.297116 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:01:02.297119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:01:02.297283 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:01:02.297403 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.297411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.298493 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.298496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.299263 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:01:02.299273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.299927 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.299932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:01:02.301022 (+ 1090us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.301440 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:01:02.301453 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:01:02.302838 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:01:02.305261 (+ 2423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.305265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.305267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.305521 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.305853 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:01:02.305858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:01:02.305863 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:01:02.305956 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:01:02.307087 (+ 1131us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:01:02.307092 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:01:02.307198 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 23:01:02.307290 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 23:01:02.307700 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":229}
I20260810 23:01:02.308550 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.295197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.29.78.188:35769)
0810 23:01:02.296324 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.296330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.296341 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:01:02.296392 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.296395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.296436 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.296520 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.297518 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.298379 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.299377 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.299815 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.300195 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.302978 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.302983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.302986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.303024 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.305151 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.305634 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.305637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.305639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.305719 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.306048 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.306053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.306057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.306717 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.308143 (+ 1426us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.308218 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.308352 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":276,"server-negotiator.queue_time_us":1077,"spinlock_wait_cycles":5376,"thread_start_us":124,"threads_started":1}
I20260810 23:01:02.310027 23100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: Bootstrap starting.
I20260810 23:01:02.310155 23012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:02.310531 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.297081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44726 (local address 127.29.78.189:39255)
0810 23:01:02.297176 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.297180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.298033 (+ 853us) server_negotiation.cc:407] Connection header received
0810 23:01:02.298206 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.298209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.298248 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.298293 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.299114 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.300556 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.301947 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.302326 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.302403 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.305975 (+ 3572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.305982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.305988 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.306018 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.307707 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.308212 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.308215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.308217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.308316 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.308648 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.308652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.308654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.308880 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.309077 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.310250 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.310374 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":40}
I20260810 23:01:02.311039 23012 log.cc:824] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:02.311153 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: Bootstrap starting.
I20260810 23:01:02.312655 23100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:02.313191 23190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: Neither blocks nor log segments found. Creating new log.
I20260810 23:01:02.313555 23100 log.cc:824] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:02.313907 23012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316: No bootstrap required, opened a new log
I20260810 23:01:02.313979 23190 log.cc:824] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:01:02.315434 23100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: No bootstrap required, opened a new log
I20260810 23:01:02.316108 23190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: No bootstrap required, opened a new log
I20260810 23:01:02.319160 23100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.319437 23190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.319701 23190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6089838641421ba3bada08ba88096a, State: Initialized, Role: FOLLOWER
I20260810 23:01:02.319716 23100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4c08c67d724bc2a7920eda29998f2b, State: Initialized, Role: FOLLOWER
I20260810 23:01:02.320125 23100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.320127 23190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.321475 23190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:02.321503 23100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:02.322415 30010 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:01:02.322963 23012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.323195 23012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3d0d4795104adc8981eb333346e316, State: Initialized, Role: FOLLOWER
I20260810 23:01:02.323561 23012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.323679 22981 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1d3d0d4795104adc8981eb333346e316"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57460
I20260810 23:01:02.324982 23012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:01:02.325067 22981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:01:02.325201 22981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:02.326557 22981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.327780 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.327903 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:02.325542 23020 master_runner.cc:186] Error getting master registration for 127.29.78.190:39429: OK, instance_id { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" instance_seqno: 1786402861825850 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:01:02.328939 23020 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:01:02.328883 23071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d3d0d4795104adc8981eb333346e316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8e4c08c67d724bc2a7920eda29998f2b"
I20260810 23:01:02.329106 23071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:02.330334 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.330421 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:02.330451 23071 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3d0d4795104adc8981eb333346e316 in term 1.
W20260810 23:01:02.330839 23228 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:02.330885 23228 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:02.330857 23158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1d3d0d4795104adc8981eb333346e316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1b6089838641421ba3bada08ba88096a"
I20260810 23:01:02.331053 23158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:01:02.332335 23158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d3d0d4795104adc8981eb333346e316 in term 1.
I20260810 23:01:02.327459 22981 leader_election.cc:288] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [CANDIDATE]: Term 1 election: Requested vote from peers 8e4c08c67d724bc2a7920eda29998f2b (127.29.78.189:39255), 1b6089838641421ba3bada08ba88096a (127.29.78.188:35769)
I20260810 23:01:02.334810 22948 leader_election.cc:302] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6089838641421ba3bada08ba88096a, 1d3d0d4795104adc8981eb333346e316; no voters:
I20260810 23:01:02.328037 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.336407 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:02.336505 23216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:01:02.336570 23216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [term 1 LEADER]: Becoming Leader. State: Replica: 1d3d0d4795104adc8981eb333346e316, State: Running, Role: LEADER
I20260810 23:01:02.336855 23216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } }
I20260810 23:01:02.338594 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3d0d4795104adc8981eb333346e316. Latest consensus state: current_term: 1 leader_uuid: "1d3d0d4795104adc8981eb333346e316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.338690 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: This master's current role is: LEADER
W20260810 23:01:02.338737 23221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:02.338804 23221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:01:02.330125 23239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:01:02.338876 23239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:01:02.341445 22923 master_runner.cc:263] No leader master found from master 1d3d0d4795104adc8981eb333346e316
I20260810 23:01:02.341521 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:01:02.342953 23109 master_runner.cc:263] No leader master found from master 1b6089838641421ba3bada08ba88096a
I20260810 23:01:02.343032 23109 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:01:02.347491 23252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:01:02.348402 23252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:01:02.351629 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.339761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57464 (local address 127.29.78.190:39429)
0810 23:01:02.339889 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.339895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.341500 (+ 1605us) server_negotiation.cc:407] Connection header received
0810 23:01:02.341683 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.341686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.341731 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.341782 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.343814 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.344572 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.345658 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.345997 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.346068 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.347954 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.347958 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.347961 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.348004 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.349775 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.350300 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.350303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.350305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.350388 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.350678 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.350681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.350683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.350884 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.350993 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.351340 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.351457 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:01:02.367183 23158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [term 1 FOLLOWER]: Refusing update from remote peer 1d3d0d4795104adc8981eb333346e316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:02.368005 23250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:02.370169 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.341072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.29.78.189:39255)
0810 23:01:02.341183 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.341188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.353256 (+ 12068us) server_negotiation.cc:407] Connection header received
0810 23:01:02.353420 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.353423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.353476 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.353526 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.357270 (+ 3744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.358033 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.360239 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.360578 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.360654 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.363314 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.363318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.363321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.363347 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.365010 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.367756 (+ 2746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.367760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.367762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.367848 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.368499 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.368502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.368504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.368735 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.368845 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.369862 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.370005 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:01:02.372200 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:02.340905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34620 (local address 127.29.78.188:35769)
0810 23:01:02.341017 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.341022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.354312 (+ 13290us) server_negotiation.cc:407] Connection header received
0810 23:01:02.354440 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.354443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.354493 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.354563 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.355415 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.356159 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.359131 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:02.359463 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:02.359561 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:02.365520 (+ 5959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:01:02.365524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:01:02.365527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:01:02.365555 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:01:02.367287 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.368145 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.368149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.368150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.368225 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:01:02.369146 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:01:02.369149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:01:02.369151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:01:02.369359 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:01:02.369462 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:02.371916 (+ 2454us) server_negotiation.cc:299] Negotiation successful
0810 23:01:02.372051 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:01:02.372430 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3d0d4795104adc8981eb333346e316. Latest consensus state: current_term: 1 leader_uuid: "1d3d0d4795104adc8981eb333346e316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.372574 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:02.373556 23071 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [term 1 FOLLOWER]: Refusing update from remote peer 1d3d0d4795104adc8981eb333346e316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:01:02.374157 23250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:01:02.377751 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d3d0d4795104adc8981eb333346e316. Latest consensus state: current_term: 1 leader_uuid: "1d3d0d4795104adc8981eb333346e316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.377913 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:02.382400 23250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3d0d4795104adc8981eb333346e316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.382530 23250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:02.383510 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3d0d4795104adc8981eb333346e316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.383630 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:02.385914 23257 mvcc.cc:205] Tried to move back new op lower bound from 7317106124208435200 to 7317106124135034880. Current Snapshot: MvccSnapshot[applied={T|T < 7317106124208435200}]
I20260810 23:01:02.389732 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d3d0d4795104adc8981eb333346e316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.389868 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316 [sys.catalog]: This master's current role is: LEADER
I20260810 23:01:02.390954 23200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d3d0d4795104adc8981eb333346e316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d3d0d4795104adc8981eb333346e316" member_type: VOTER last_known_addr { host: "127.29.78.190" port: 39429 } } peers { permanent_uuid: "8e4c08c67d724bc2a7920eda29998f2b" member_type: VOTER last_known_addr { host: "127.29.78.189" port: 39255 } } peers { permanent_uuid: "1b6089838641421ba3bada08ba88096a" member_type: VOTER last_known_addr { host: "127.29.78.188" port: 35769 } } }
I20260810 23:01:02.391081 23200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:01:02.391572 23260 mvcc.cc:205] Tried to move back new op lower bound from 7317106124208435200 to 7317106124135034880. Current Snapshot: MvccSnapshot[applied={T|T < 7317106124208435200}]
I20260810 23:01:02.393102 23252 catalog_manager.cc:1347] Generated new cluster ID: dc1400da37004113ae860266afb5adf9
I20260810 23:01:02.393160 23252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:01:02.417413 23252 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:01:02.420725 23252 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:01:02.441932 23252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d3d0d4795104adc8981eb333346e316: Generated new TSK 0
I20260810 23:01:02.442868 23252 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:01:02.473898 23014 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.470842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.29.78.190:39429)
0810 23:01:02.470965 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.470970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.472829 (+ 1859us) server_negotiation.cc:407] Connection header received
0810 23:01:02.473053 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.473056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.473105 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.473155 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.473749 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57478: BlockingRecv error: recv got EOF from 127.0.0.1:57478 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260810 23:01:02.475478 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.471558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44746 (local address 127.29.78.189:39255)
0810 23:01:02.471702 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.471707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.474595 (+ 2888us) server_negotiation.cc:407] Connection header received
0810 23:01:02.474768 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.474772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.474820 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.474871 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.475375 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44746: BlockingRecv error: recv got EOF from 127.0.0.1:44746 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 23:01:02.477162 23193 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.472257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.29.78.188:35769)
0810 23:01:02.472361 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.472365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.475997 (+ 3632us) server_negotiation.cc:407] Connection header received
0810 23:01:02.476419 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.476423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.476475 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.476529 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.477054 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34630: BlockingRecv error: recv got EOF from 127.0.0.1:34630 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 23:01:02.481756 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.480208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.29.78.189:39255)
0810 23:01:02.480311 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.480315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.480790 (+ 475us) server_negotiation.cc:407] Connection header received
0810 23:01:02.481013 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.481017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.481087 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.481141 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.481648 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44758: BlockingRecv error: recv got EOF from 127.0.0.1:44758 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 23:01:02.487123 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.485581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.29.78.189:39255)
0810 23:01:02.485679 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.485683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.486202 (+ 519us) server_negotiation.cc:407] Connection header received
0810 23:01:02.486388 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.486393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.486460 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.486514 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.487025 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44766: BlockingRecv error: recv got EOF from 127.0.0.1:44766 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 23:01:02.492970 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.491416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.29.78.189:39255)
0810 23:01:02.491516 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.491520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.492058 (+ 538us) server_negotiation.cc:407] Connection header received
0810 23:01:02.492251 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.492256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.492310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.492359 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.492855 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44778: BlockingRecv error: recv got EOF from 127.0.0.1:44778 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 23:01:02.501111 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.499315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.29.78.189:39255)
0810 23:01:02.499442 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.499446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.499997 (+ 551us) server_negotiation.cc:407] Connection header received
0810 23:01:02.500302 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.500306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.500361 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.500415 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.501015 (+ 600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44784: BlockingRecv error: recv got EOF from 127.0.0.1:44784 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 23:01:02.513437 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.511764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.29.78.189:39255)
0810 23:01:02.511860 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.511864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.512368 (+ 504us) server_negotiation.cc:407] Connection header received
0810 23:01:02.512658 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.512662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.512714 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.512768 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.513341 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44794: BlockingRecv error: recv got EOF from 127.0.0.1:44794 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 23:01:02.533648 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.532050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.29.78.189:39255)
0810 23:01:02.532177 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.532182 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.532702 (+ 520us) server_negotiation.cc:407] Connection header received
0810 23:01:02.532896 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.532899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.532947 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.532994 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.533540 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44802: BlockingRecv error: recv got EOF from 127.0.0.1:44802 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 23:01:02.569394 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.568209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.29.78.189:39255)
0810 23:01:02.568352 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.568356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.568507 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:01:02.568624 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.568626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.568673 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.568745 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.569234 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44806: BlockingRecv error: recv got EOF from 127.0.0.1:44806 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 23:01:02.636940 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.635702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44810 (local address 127.29.78.189:39255)
0810 23:01:02.635877 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.635882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.636031 (+ 149us) server_negotiation.cc:407] Connection header received
0810 23:01:02.636212 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.636220 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.636269 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.636361 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.636833 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44810: BlockingRecv error: recv got EOF from 127.0.0.1:44810 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 23:01:02.768637 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:02.767456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.29.78.189:39255)
0810 23:01:02.767621 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:02.767626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:02.767753 (+ 127us) server_negotiation.cc:407] Connection header received
0810 23:01:02.767925 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:02.767928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:02.767995 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:02.768086 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:02.768526 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44816: BlockingRecv error: recv got EOF from 127.0.0.1:44816 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 23:01:03.028270 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:01:03.027127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.29.78.189:39255)
0810 23:01:03.027276 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:03.027283 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:03.027423 (+ 140us) server_negotiation.cc:407] Connection header received
0810 23:01:03.027556 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:03.027558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:03.027608 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:03.027682 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:01:03.028139 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44824: BlockingRecv error: recv got EOF from 127.0.0.1:44824 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:01:03.332264 23228 catalog_manager.cc:1259] Loaded cluster ID: dc1400da37004113ae860266afb5adf9
I20260810 23:01:03.332386 23228 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: loading cluster ID for follower catalog manager: success
I20260810 23:01:03.334048 23228 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: acquiring CA information for follower catalog manager: success
I20260810 23:01:03.335147 23228 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1b6089838641421ba3bada08ba88096a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:01:03.339938 23221 catalog_manager.cc:1259] Loaded cluster ID: dc1400da37004113ae860266afb5adf9
I20260810 23:01:03.339999 23221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: loading cluster ID for follower catalog manager: success
I20260810 23:01:03.341485 23221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: acquiring CA information for follower catalog manager: success
I20260810 23:01:03.342514 23221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e4c08c67d724bc2a7920eda29998f2b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:01:03.545754 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:03.542275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.29.78.189:39255)
0810 23:01:03.542510 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:03.542513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:03.542527 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:03.542649 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:03.542651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:03.542768 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:03.542859 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:03.543081 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.543578 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.544382 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.544619 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.544695 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:03.544948 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:03.545359 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:03.545449 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:03.545506 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 23:01:03.545551 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260810 23:01:03.551784 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:03.548525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.29.78.188:35769)
0810 23:01:03.548746 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:03.548750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:03.548778 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:01:03.548847 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:03.548850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:03.548946 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:03.549035 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:03.549244 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.549737 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.550629 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.550846 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.550923 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:03.551151 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:03.551474 (+ 323us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:03.551559 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:03.551634 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 23:01:03.551679 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":83,"threads_started":1}
I20260810 23:01:03.558293 23351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:03.554468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.29.78.188:35769)
0810 23:01:03.554564 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:03.554568 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:03.554933 (+ 365us) server_negotiation.cc:407] Connection header received
0810 23:01:03.555034 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:03.555036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:03.555100 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:03.555172 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:03.555454 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.556216 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.557190 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.557448 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.557543 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:03.557788 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:03.557890 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:03.557969 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:03.558104 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:01:03.558163 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:01:03.558712 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:03.554726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44834 (local address 127.29.78.189:39255)
0810 23:01:03.554823 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:03.554825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:03.555042 (+ 217us) server_negotiation.cc:407] Connection header received
0810 23:01:03.555234 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:03.555239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:03.555291 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:03.555335 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:03.555706 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.556160 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.557393 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.557693 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.557805 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:03.558215 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:03.558284 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:03.558332 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:03.558549 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 23:01:03.558600 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:01:03.559330 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:01:03.554459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57488 (local address 127.29.78.190:39429)
0810 23:01:03.554930 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0810 23:01:03.554934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:01:03.554948 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:01:03.555037 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:01:03.555040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:01:03.555199 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:01:03.555314 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:01:03.555542 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.556351 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.557464 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:01:03.557894 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:01:03.558010 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:01:03.558434 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:01:03.558703 (+ 269us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:01:03.558793 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:01:03.559096 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 23:01:03.559172 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":57,"threads_started":1}
I20260810 23:01:03.562371 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 22923
I20260810 23:01:03.574841 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 23020
I20260810 23:01:03.582811 30010 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskX35aNT/build/debug/bin/kudu with pid 23109
2026-08-10T23:01:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3099 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6174 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284770 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST